Research map / Large language models / AI and society
Predicting Empirical AI Research Outcomes with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a benchmark of 1,585 test (and 6,000 train) idea pairs scraped from AI conference papers and a GPT-4.1 fine-tuned system with agentic paper retrieval that predicts which research idea empirically wins, outperforming human experts in NLP.
Problem
Predicting which of two AI research ideas will perform better empirically, without implementing them, is costly to do manually and has no established benchmark or strong baselines.
Approach
Frame research outcome prediction as pairwise comparison of idea summaries labeled from published results, and combine a fine-tuned GPT-4.1 with an agentic retrieval module that summarizes full retrieved papers (pre-cutoff) to predict the winner.
Key results
- Fine-tuned GPT-4.1 with retrieval reaches 77% accuracy on the 1,585-example test set, versus 51.9% for zero-shot GPT-4.1 and near-chance for o3.
- On 45 NLP idea pairs, the system achieves 64.4% vs. 48.9% for majority-vote of 5 NLP experts.
- On 33 unpublished ideas (including AI-generated ones), the system achieves 63.6% accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune GPT-4.1 on historical idea pairs | GPT-4.1 · Full test set | accuracy: 51.9% to 77% | Helped |
| Fine-tune GPT-3.5-turbo on historical idea pairs | GPT-3.5-turbo · Full test set | accuracy: 45.0% to 53.7% | Helped |
| Summarize retrieved full papers instead of using abstracts (ablation) | GPT-4.1 with RAG · 600 test examples | accuracy: 38.8 to 53.0 | Helped |
| Use abstracts rather than summaries of retrieved full papers (ablation) | GPT-4.1 with RAG · 600 test examples | accuracy: 53.0 to 38.8 | Hurt |
| Add self-augmented chain-of-thought training with randomly sampled CoTs (ablation) | Fine-tuned GPT-4.1 · Full test set | accuracy | No clear effect |
| Add self-augmented chain-of-thought training with LM-selected CoTs (ablation) | Fine-tuned GPT-4.1 · Full test set | accuracy | No clear effect |
Surprising findings
- Off-the-shelf frontier models such as o3 perform no better than random guessing even with the same retrieval augmentation.
- Self-generated chain-of-thought fine-tuning yields no gain over plain fine-tuning on this task.
Limitations
- System is a black box, limiting human-AI collaboration and interpretability.
- Cannot fully rule out reliance on spurious features despite robustness tests.
- Novel-idea evaluation is small (33 pairs) and limited to prompting-method topics.
Experimental setup
Datasets: ACL EMNLP NAACL COLM NeurIPS ICLR CVPR ECCV ACMMM CoRL papers (scraped idea pairs)
Benchmarks: Idea-pair outcome prediction test set (1,585 human-verified examples) NLP expert subset (45 pairs) Unpublished ideas (33 pairs)
Metrics: accuracy
Baselines: GPT-4.1 zero-shot o3 o1 Claude 3.5 Sonnet GPT-3.5-turbo fine-tuned human experts (majority vote) human ceiling (best-per-topic)
Builds on: GPT-4.1 fine-tuning Exa neural search self-generated chain-of-thought fine-tuning
Closest papers by meaning
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.89)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.88)
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.87)
- AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text (ICLR 2025 · similarity 0.87)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.86)
- MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.