Research map / Large language models / Agents, code and math
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
Minju Seo, Jinheon Baek, Seongyun Lee, Sung Ju Hwang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use full paper instead of only the abstract (ablation) | one-shot code generation · Paper2CodeBench ICLR (reference-based) | score (1-5): 2.28 -> 3.08 | Improved |
| add planning, analysis and staged generation over one-shot full-paper generation | PaperCoder · Paper2CodeBench ICLR (reference-based) | score (1-5): 3.08 -> 3.68 | Improved |
| use PaperCoder instead of ChatDev | ChatDev · Paper2CodeBench NeurIPS (reference-based) | score (1-5): 2.96 -> 3.83 | Improved |
| PaperCoder compared with official repositories | PaperCoder · Paper2CodeBench ICLR (reference-free) | score (1-5): 4.84 vs 4.73 | No clear effect |
| remove the methodology section from the input paper (ablation) | PaperCoder · Paper2CodeBench subsample | reference-based score (1-5): 4.26 -> 3.75 | Hurt |
| use PaperCoder instead of IterativeAgent | IterativeAgent (o3-mini) · PaperBench Code-Dev | replication score (%): 16.4 -> 45.14 | Improved |
| use PaperCoder with Claude 3.5 Sonnet instead of IterativeAgent | IterativeAgent (Claude 3.5 Sonnet) · PaperBench Code-Dev | replication score (%): 27.5 -> 51.14 | Improved |
| use a stronger backbone (o3 instead of o3-mini) (ablation) | PaperCoder · PaperBench Code-Dev | replication score (%): 45.14 -> 60.86 | Improved |
| use o1 backbone instead of o3-mini (ablation) | PaperCoder · PaperBench Code-Dev | replication score (%): 45.14 -> 38.31 | Hurt |
| use o3-mini-high rather than GPT-4o as evaluator (ablation) | LLM judge · Paper2CodeBench | rank correlation with human: 0.74 -> 0.78 | Improved |
| add architecture design module alone in the incremental ablation (ablation) | PaperCoder · Paper2CodeBench ICML | score | Hurt |
| add verification-and-refinement to planning and analysis (ablation) | PaperCoder · Paper2CodeBench subsample (30 papers) | downstream score | Improved |
Closest papers by meaning
- ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code (NeurIPS 2025 · similarity 0.87)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.87)
- From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.86)
- PaperBench: Evaluating AI’s Ability to Replicate AI Research (ICML 2025 · similarity 0.85)
- AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data (NeurIPS 2024 · similarity 0.85)
- CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
