Research map / Large language models / AI and society
ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ResearchCodeBench, a 212-task benchmark where LLMs fill in masked code from 20 recent ML papers, scored by execution-based tests with a LoC-weighted pass rate.
Problem
Whether LLMs can faithfully implement novel ML research ideas from recent papers (unseen in pretraining) as executable code is unclear, and existing evaluations often rely on subjective LLM judging.
Approach
Fill-in-the-blank code completion tasks derived from core contributions of recent ML papers, annotated with XML-style snippet tags and hints, evaluated with human-written equivalence and unit tests, and scored by a lines-of-code-weighted scaled pass rate.
Key results
- Gemini-2.5-Pro-Preview achieves the best scaled pass rate at 37.3%, followed by O3 (High) at 32.3% and O4-mini (High) at 30.8%.
- All models show a performance drop on the 13-paper contamination-safe subset compared with the full 20-paper benchmark.
- Functional errors account for 58.6% of failures; name, type, and syntax errors each roughly 8-9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide the full research paper as context (ablation) | Gemini-2.5-Pro and O3 (High) · ResearchCodeBench | success rate: up to 30% | Helped |
| Provide the full research paper as context (ablation) | GPT-4.1-Nano and GPT_4o_Mini · ResearchCodeBench | success rate | No clear effect |
| Provide the full research paper as context (ablation) | LLaMA-based models · ResearchCodeBench | success rate | Hurt |
| Evaluate on the contamination-safe 13-paper subset instead of the full benchmark | All evaluated models · ResearchCodeBench contamination-safe 13-paper subset | Scaled Pass@1 | Hurt |
Surprising findings
- All evaluated LLaMA-based models perform worse when given the full paper as context.
- Smaller models such as GPT-4.1-Nano and GPT-4o-Mini show little to no benefit from paper context.
Limitations
- Only 20 papers, all in machine learning, limiting coverage.
- Tests are manually written, limiting scalability.
- No human baseline is provided.
Experimental setup
Datasets: 20 recent ML papers and their official repositories
Benchmarks: ResearchCodeBench
Metrics: scaled pass@1 pass@1 pass rate
Baselines: Gemini-2.5-Pro-Preview O3 (High) O4-mini (High) GPT-4.1 Claude 3.5 Sonnet GPT-4o-Mini
Builds on: HumanEval SciCode PaperBench
Closest papers by meaning
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.89)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.89)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.88)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.88)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.88)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.