Research map / Large language models / Agents, code and math
The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Automated LLM Speedrunning Benchmark, which asks AI research agents to reproduce successive NanoGPT speedrun records given hints of varying detail, and finds that current agents recover only a small fraction of human speedups.
Problem
Whether AI research agents can faithfully reproduce sequential, code-level LLM training innovations (NanoGPT speedrun records) from descriptions of the changes.
Approach
Turns each consecutive pair of NanoGPT speedrun records into a reproduction task, scored by fraction of speedup recovered (FSR) in wall-clock training time on a fixed 8xH100 node, with four hint levels (none, pseudocode, text, mini-paper) and a flexible AIDE-based tree search scaffold.
Key results
- Without hints, all agents recover less than 20% of the human speedup on average in mean FSR.
- o3-mini with pseudocode hints (flat and tree scaffolds) recovers roughly 40-43% of speedup; with pseudocode+text+mini-paper, multi-AIDE reaches 0.46 FSR.
- Gemini-2.5-Pro and Claude-3.7-Sonnet reach IQM close to 0 FSR, below open-weights DeepSeek-R1 in aggregate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide L1 pseudocode hints instead of L2 text or L3 mini-paper hints | o3-mini research agents · NanoGPT speedrun record transitions | FSR: 0.40$\pm$0.02 (Flat, L1); 0.22$\pm$0.04 (Flat, L2); 0.17$\pm$0.03 (Flat, L3) | Helped |
| Combine L1, L2, and L3 hints rather than use L1 alone | DeepSeek-R1 research agents · NanoGPT speedrun record transitions | FSR: 0.13$\pm$0.03 (Flat, L1) -> 0.30$\pm$0.04 (Flat, L1+L2+L3) | Helped |
| Combine L1+L2+L3 hints rather than L1 alone | o3-mini + Multi-AIDE · NanoGPT speedrun record transitions | FSR: 0.43$\pm$0.02 -> 0.46$\pm$0.04 | Mixed |
| Add text and mini-paper hints to L1 pseudocode | o3-mini research agents · NanoGPT speedrun record transitions | FSR: 0.40$\pm$0.02 (Flat, L1) -> 0.24$\pm$0.05 (Flat, L1+L2+L3) | Hurt |
| Add FlexAttention documentation to the model context | DeepSeek-R1 · Record 12 (FlexAttention) | FSR: 0.09$\pm$0.01 -> 0.07$\pm$0.01 | Hurt |
| Add FlexAttention documentation to the model context | o3-mini · Record 12 (FlexAttention) | FSR: 0.10$\pm$0.01 -> 0.06$\pm$0.01 | Hurt |
| Use flat best-of-M search rather than iterative search scaffolds | LLM research agents · NanoGPT speedrun record transitions with hints | FSR | Mixed |
| Use explicit debugging steps in the search scaffold | LLM research agents · NanoGPT speedrun record transitions | FSR | No clear effect |
| Use branching search instead of flat search | LLM research agents · NanoGPT speedrun record transitions | Buggy-node proportion | Helped |
| Start a cumulative task from agent-generated prior code instead of ground-truth prior code | o3-mini + Multi-AIDE with L1+L2+L3 · Cumulative speedrun | FSR: approximately 60%; around 20%; any speedup | Hurt |
Surprising findings
- Adding FlexAttention documentation as a hint worsened performance on record 12 for both DeepSeek-R1 and o3-mini.
- DeepSeek-R1 agents often performed worse with individual hints than with no hints, frequently producing buggy code.
Limitations
- Hints are manually curated and succinct, so the benchmark does not test autonomous external knowledge retrieval.
- Possible memorization of speedrun records by models with knowledge cutoffs after the record publication dates.
- Tests reproduction rather than innovation and is limited to a single-node LLM training setting.
Experimental setup
Datasets: NanoGPT speedrun records (19 record transitions) FineWeb
Benchmarks: Automated LLM Speedrunning Benchmark
Metrics: fraction of speedup recovered (FSR) interquartile mean (IQM) L2 embedding distance recovered LLM judge reproducibility score
Baselines: MLE-bench PaperBench CORE-bench RE-bench MLAgentBench MLGym-bench AIDE
Builds on: AIDE Aider Muon NanoGPT speedrun
Closest papers by meaning
- Steering Large Language Models between Code Execution and Textual Reasoning (ICLR 2025 · similarity 0.90)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.89)
- LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling (NeurIPS 2026 · similarity 0.89)
- AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks (NeurIPS 2025 · similarity 0.89)
- Fast and Accurate Probing of In-Training LLMs' Downstream Performances (NeurIPS 2026 · similarity 0.89)
- Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.