Research map / Large language models / AI and society
AI-Researcher: Autonomous Scientific Innovation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AI-Researcher, a multi-agent system that automates literature review, idea generation, code implementation, and paper writing, and Scientist-Bench, a 22-paper benchmark for evaluating it with LLM judges.
Problem
Autonomous scientific research (idea-to-paper) lacks end-to-end systems and standardized benchmarks for measuring implementation success and research quality against human papers.
Approach
A multi-agent pipeline (Knowledge Acquisition, Resource Analyst, Idea Generator, Code Agent, Advisor Agent, Documentation Agent) with iterative advisor feedback and hierarchical manuscript writing, evaluated by pairwise LLM review against human-authored papers, using Level-1 (guided) and Level-2 (open-ended) tasks.
Key results
- Claude-series models achieved 93.8% implementation completeness across the benchmark, versus 50% for 4o-series on a subset (87.5% vs 50% for Claude).
- Overall AI-generated papers averaged about -0.58 to -1.76 ratings vs human papers, with 15.79% to 78.95% judged comparable depending on evaluator.
- In open-ended Level-2 tasks, comparable rates rose to 40% to 100%, with average ratings from -0.20 to -1.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use open-ended exploration without explicit research instructions rather than guided Level-1 tasks | AI-Researcher · Scientist-Bench scientific paper evaluation | Average Rating: -0.58~-1.76 to -0.20~-1.01 | Helped |
| use open-ended exploration without explicit research instructions rather than guided Level-1 tasks | AI-Researcher · Scientist-Bench scientific paper evaluation | Comparable(%): 15.79%~78.95% to 40.00%~100.00% | Helped |
| use Claude-series instead of 4o-series as the research-agent backbone (ablation) | AI-Researcher · Scientist-Bench evaluation subset | completeness: 50% -> 87.5% | Helped |
| use Claude-series instead of 4o-series as the research-agent backbone (ablation) | AI-Researcher · Scientist-Bench evaluation subset | correctness: 1.0 -> 2.75 | Helped |
| perform Level-2 novel-idea implementation instead of Level-1 adaptation | AI-Researcher (Claude-series) · Scientist-Bench implementation | correctness: 2.5 -> 2.25 | Hurt |
| use Claude-3.5 rather than GPT-4o as the research-agent backbone (ablation) | AI-Researcher · Scientist-Bench backbone comparison | Comparable(%): 71.43% -> 85.71% (GPT-4o reviewer); 42.86% -> 28.57% (o1-mini reviewer); 0.00% -> 14.29% (o3-mini reviewer); 0.00% -> 0. | Mixed |
Surprising findings
- Open-ended Level-2 exploration outperformed guided Level-1 implementation tasks on review ratings.
- GPT-4o claimed a Diffusion Transformer implementation but delivered only a standard ViT without diffusion components.
Limitations
- Generated papers remain below human quality on average, with weak theoretical analysis and limited use of advanced methods.
- Evaluation relies on LLM judges with notable divergence across evaluators and small benchmark (22 papers, 5-6 Level-2 tasks).
Experimental setup
Datasets: Scientist-Bench (22 papers from 2022-2024 across diffusion models, vector quantization, GNNs, recommender systems) ICLR 2021-2023 accept/reject paper pairs (32 pairs)
Benchmarks: Scientist-Bench ICLR 2021-2023 accept/reject pairs
Metrics: completeness rate correctness score (1-5) mean pairwise rating (-3 to 3) comparable percentage (rating >= -1.0) selection accuracy
Baselines: AI Scientist Agent Laboratory CycleResearcher AI co-scientist
Builds on: AI Scientist Agent Laboratory MetaGPT AutoGen ResearchAgent
Closest papers by meaning
- Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges (AAAI 2025 · similarity 0.92)
- Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery (NeurIPS 2026 · similarity 0.91)
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.89)
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.89)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.89)
- From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.