Research map / Large language models / Agents, code and math
MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MLRC-Bench, a dynamic benchmark of 7 ML conference competitions to evaluate whether language agents can propose and implement novel research methods, using objective metrics and checking LLM-as-judge novelty scores against performance.
Problem
Existing evaluations of LLM research agents either judge end-to-end papers subjectively or measure Kaggle-style code generation without requiring methodological novelty; there is no objective benchmark for proposing and implementing novel ML research methods.
Approach
Sources tasks from recent ML competitions with reliable baselines and top human solutions, standardizes them into a repository-level, agent-agnostic environment with compute constraints and dev/test splits, and normalizes scores as relative improvement to the human solution. It also tests whether LLM-judged innovativeness correlates with objective effectiveness.
Key results
- Best agent (gemini-exp-1206 under MLAB) closes only 9.3% of the gap between baseline and top human score on average across 7 tasks.
- Claude 3.5 Sonnet v2 under MLAB achieves the best result on several tasks but scores -94.7 on machine unlearning, degrading below baseline.
- LLM-judged innovativeness shows near-zero correlation with effectiveness (-0.06), indicating LLM-judged novelty is not a reliable proxy for performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide additional human- or AI-generated ideas to MLAB | GPT-4o with MLAB · MLRC-Bench tasks | relative improvement to human solution: MLAB (GPT-4o) Avg 5.4; w/ Human Idea Avg 3.5; w/ CoI-Agent Idea (o1) Avg 7.1 | Mixed |
| Provide a human-curated idea before implementation | MLAB · Backdoor Trigger Recovery | Pass@1: 0.12 -> 0.31 | Helped |
| Provide a human-curated idea before implementation | MLAB · Machine Unlearning | Pass@1: 0.00 -> 0.12 | Helped |
| Provide a CoI-Agent-generated idea before implementation | MLAB · Machine Unlearning | Pass@1: 0.00 -> 0.06 | Helped |
| Use an LLM judge's innovativeness rating as a proxy for effectiveness | Agent-generated solutions across tasks · MLRC-Bench | Spearman correlation: -0.06 | No clear effect |
Surprising findings
- Adding human or AI-generated ideas does not consistently improve agent performance over MLAB alone.
- LLM judges rate agent solutions more favorably than objective metrics do, creating an overly optimistic picture.
Limitations
- Only seven competitions are included so far.
- Evaluation uses only the general-purpose MLAB agent scaffold, as other frameworks did not suit repository-level tasks.
- Only 8 trials per configuration due to API budget, reporting best attempt.
Experimental setup
Datasets: LLM Merging (NeurIPS 2024) Backdoor Trigger Recovery (NeurIPS 2024) Temporal Action Localisation (ECCV 2024 workshop) Rainfall Prediction (NeurIPS 2023) Machine Unlearning (NeurIPS 2023) Next Product Recommendation (KDD Cup 2023) Cross-Domain Meta Learning (NeurIPS 2022)
Benchmarks: MLRC-Bench
Metrics: relative improvement to human solution pass@k lines of code runtime Spearman correlation
Baselines: MLAB (claude-3-5-sonnet-v2) MLAB (llama3-1-405b-instruct) MLAB (o3-mini) MLAB (gpt-4o) CoI-Agent Idea + MLAB Human Idea + MLAB
Builds on: MLAB Chain-of-Ideas MLE-Bench ReAct
Closest papers by meaning
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.93)
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (ICLR 2025 · similarity 0.91)
- \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.90)
- AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench (NeurIPS 2025 · similarity 0.89)
- MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation (ICML 2024 · similarity 0.89)
- MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.