Research map / Large language models / AI and society
AAAR-1.0: Assessing AI’s Potential to Assist Research
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AAAR-1.0, an expert-annotated benchmark of four research-assistance tasks (equation inference, experiment design, paper weakness identification, review deficiency critique) and evaluates open- and closed-source LLMs on them.
Problem
Existing LLM evaluations focus on everyday tasks; there is no systematic, single-step benchmark assessing whether LLMs can perform expertise-intensive research tasks such as checking equations, designing experiments, finding paper weaknesses, and critiquing reviews.
Approach
Builds four singular, expert-curated tasks with clear input-output formats and task-specific automatic metrics (LLM-judged entailment F1, semantic similarity S-Match/S-Precision/S-Recall, and a novel ITF-IDF diversity metric), rather than end-to-end research agents.
Key results
- On EquationInference, the best model (o3-mini) reaches 47.98 F1 versus 40.00 for the all-positive baseline, while most open-source LLMs fall below it.
- On ExperimentDesign, closed-source LLMs score about 20-30 En-F1; o1-preview achieves the best S-Match of 58.55 for explanations, and human acceptance ratio ranks o1-preview highest (76.14%), with Spearman correlation 1 to S-Match ranking.
- On PaperWeakness, GPT-4o with split-combine achieves 47.73 S-F1 and 5.95 ITF-IDF, while the AI-Scientist agent scores lower (45.05 S-F1, 2.23 ITF-IDF) than its GPT-4o backbone; human reviews reach 7.69 ITF-IDF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale up input context length (ablation) | LLMs on EquationInference · EquationInference | F1 | Mixed |
| provide the complete experiment list instead of explaining experiments one by one (ablation) | open-source LLMs on Experiment explanation · ExperimentDesign | S-Match: 50.05 -> 49.36; 51.12 -> 48.56 | Hurt |
| provide the complete experiment list instead of explaining experiments one by one (ablation) | closed-source LLMs on Experiment explanation · ExperimentDesign | S-Match: 52.87 -> 57.48; 53.03 -> 59.11; 54.79 -> 58.54; 58.55 -> 61.58 | Helped |
| add figure inputs (ablation) | GPT-4o on ExperimentDesign · ExperimentDesign | En-F1: 25.03 -> 25.39 | No clear effect |
| add figure inputs (ablation) | InternVL2-26B on ExperimentDesign · ExperimentDesign | En-F1: 24.26 -> 15.04 | Hurt |
| use split-combine processing instead of no-split input (ablation) | GPT-4o on PaperWeakness · PaperWeakness | S-F1: 45.74 -> 47.73; 45.47 -> 47.73 | Helped |
| use split-combine processing instead of no-split input (ablation) | AI-SCI on PaperWeakness · PaperWeakness | S-F1: 42.56 -> 45.05; 42.53 -> 45.05 | Helped |
| add table images (ablation) | GPT-4o on PaperWeakness · PaperWeakness | S-F1: 47.73 -> 46.76 | Hurt |
| add figure images (ablation) | GPT-4o on PaperWeakness · PaperWeakness | S-F1: 47.73 -> 46.62 | Hurt |
| add table and figure images (ablation) | GPT-4o on PaperWeakness · PaperWeakness | S-F1: 47.73 -> 46.58 | Hurt |
| add table images (ablation) | InternVL2-26B on PaperWeakness · PaperWeakness | S-F1: 41.91 -> 40.55 | Hurt |
| add figure images (ablation) | InternVL2-26B on PaperWeakness · PaperWeakness | S-F1: 41.91 -> 42.88 | Helped |
Surprising findings
- The AI-Scientist agent underperforms its own GPT-4o backbone on paper weakness identification, suggesting popular prompting techniques do not help this task.
- Adding figure or table images does not reliably improve LMM performance on experiment design or weakness identification, and sometimes hurts it.
Limitations
- Possible data leakage since source data from arXiv and OpenReview may appear in LLM pretraining corpora.
- Small dataset size for ExperimentDesign (100 instances) due to expert annotation cost, causing variability.
- ReviewCritique results are taken from prior work rather than newly run experiments.
Experimental setup
Datasets: ACL Anthology accepted papers 2019-2023 (LaTeX via arXiv) arXiv cs.AI/cs.CL/cs.CV papers 2018-2023 ICLR 2023 OpenReview submissions ReviewCritique meta-review dataset (prior work)
Benchmarks: AAAR-1.0 EquationInference AAAR-1.0 ExperimentDesign AAAR-1.0 PaperWeakness AAAR-1.0 ReviewCritique
Metrics: F1 precision recall En-Precision En-Recall S-Match ROUGE-1 ROUGE-L S-F1 S-Precision
Baselines: AI-Scientist (AI-SCI) All-Positive baseline Copy Input baseline Human Review
Builds on: AI-Scientist SentenceBERT VILA PDFFigures-2.0
Closest papers by meaning
- Predicting Empirical AI Research Outcomes with Language Models (NeurIPS 2025 · similarity 0.89)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.88)
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.88)
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.87)
- Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (ICML 2024 · similarity 0.87)
- SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.