About
What is Paper2Code?
Paper2Code is an innovative tool designed to automate the generation of code from scientific papers, specifically in the field of machine learning. At its core is PaperCoder, a sophisticated multi-agent LLM system that streamlines the process of converting research papers into functional code repositories. This system operates through a meticulously structured three-stage pipeline: planning, analysis, and code generation, with each stage handled by specialized AI agents. Paper2Code aims to significantly reduce the manual effort and time typically required to implement research findings, offering a robust solution for developers and researchers. It supports both OpenAI API and open-source models like DeepSeek-Coder-V2-Lite-Instruct, providing flexibility in deployment and cost management. The tool also includes comprehensive benchmark datasets and a model-based evaluation system for assessing the quality of generated repositories.
Best used for
Ideal for developers and researchers who need to quickly translate machine learning research papers into executable code, validate implementations against benchmarks, and assess the quality of generated repositories. Especially valuable for accelerating research prototyping and ensuring faithful reproduction of scientific findings.
Common actions
github copilot"AI Agents"face swappingcollaborationlow-code/no-codeautomated workflowdeepfakeworkflowsopen-source
Capabilities
Key features
- Automate code generation
- Multi-agent LLM system
- Three-stage pipeline
- Supports OpenAI API
- Supports open-source models
- Benchmark datasets
- Model-based evaluation
Pricing & Plans
Open Source ยท Usage-based
FAQs
What is the estimated cost for using Paper2Code with the OpenAI API?
Using the o3-mini version of the OpenAI API with Paper2Code is estimated to cost between $0.50 and $0.70 per run. This cost is for generating code from a single paper, such as 'Attention Is All You Need'.
Can Paper2Code be used with open-source language models?
Yes, Paper2Code supports open-source models, with deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct as the default. Users can run these models locally using vLLM, providing an alternative to the OpenAI API for code generation.
How does Paper2Code evaluate the quality of generated code repositories?
Paper2Code employs a model-based evaluation approach that supports both reference-free and reference-based settings. It critiques key implementation components, assigns severity levels, and generates a 1-5 correctness score, typically averaged over 8 samples using o3-mini-high.