Research map / Large language models / Agents, code and math
InnoGym: Benchmarking the Innovation Potential of AI Agents
Jintian Zhang, Kewei Xu, Jingsheng Zheng, Zhuoyun Yu, Yuqi Zhu, Yujie Luo, Lanning Wei, Shuofei Qiao, Lun Du, Da Zheng, Shumin Deng, Huajun Chen and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compare MLAB with AIDE under pessimistic imputation | agent scaffolds (DeepSeek-v3.1) · iBench (10 tasks) | normalized ratio R: -0.82 -> -0.62 (diff +0.20, p=0.007) | Improved |
| compare MLAB with AIDE on novelty | agent scaffolds (DeepSeek-v3.1) · iBench (10 tasks) | novelty: 23.33 vs 39.58 (p=0.133) | No clear effect |
| add an innovation-encouraging prompt (ablation) | AIDE · CirclePacking | novelty: 35.33 -> 58.33 | Improved |
| add an innovation-encouraging prompt (ablation) | AIDE · CirclePacking | performance gain: -0.25 -> -0.37 | Hurt |
| add an innovation-encouraging prompt (ablation) | AIDE · OAG | novelty: 70.83 -> 50.00 | Hurt |
| add an innovation-encouraging prompt (ablation) | AIDE · BEETL(Sleep) | performance gain: -53.62 -> -54.56 | Hurt |
| use stronger backbone LLM (Gemini-2.5-Pro instead of DeepSeek-v3.1) (ablation) | agent framework · CirclePacking (sum of radii) | score: 2.40 -> 2.49 | Improved |
| raise sampling temperature (ablation) | agent framework · CirclePacking | novelty vs performance gain: qualitative (Fig. only) | Mixed |
| longer execution time budget (ablation) | agent framework · CirclePacking | performance gain / novelty: qualitative (Fig. only) | Mixed |
Closest papers by meaning
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.94)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.89)
- FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics (NeurIPS 2026 · similarity 0.89)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.89)
- From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.88)
- ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
