Research map / Large language models / Agents, code and math
CiteME: Can Language Models Accurately Cite Scientific Claims?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CiteME, a human-curated benchmark of 130 unambiguous citation-attribution excerpts from recent ML papers, and CiteAgent, a GPT-4o agent with search and read tools that reaches 35.3% accuracy versus 69.7% for humans.
Problem
Language models often fail to attribute scientific claims to their correct sources, and existing citation-attribution benchmarks are automatically curated, containing many ambiguous, unattributable, or trivial excerpts and restricting retrieval to small pre-selected corpora.
Approach
A manually curated benchmark of excerpts that unambiguously reference a single paper, filtered to remove items GPT-4o answers without tools, paired with an agent that uses Semantic Scholar search and paper reading to find the cited paper open-endedly.
Key results
- CiteAgent with GPT-4o (search and read, with demonstration) reaches 35.3% accuracy on CiteME, versus 0% for SPECTER and SPECTER2.
- Human experts reach 69.7% accuracy on CiteME, averaging 38.2 seconds per instance.
- Without tools, frontier LMs score 0-18.5% on CiteME; o1-preview scores 38.7% without commands, which the authors attribute to possible memorization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add search and read commands to GPT-4o (ablation) | CiteAgent on CiteME · CiteME | accuracy: 0 -> 22.7 | Helped |
| add a demonstration to GPT-4o with Search and Read (ablation) | CiteAgent on CiteME · CiteME | accuracy: 22.7 -> 35.3 | Helped |
| add search commands to LLaMA-3-70B (ablation) | CiteAgent on CiteME · CiteME | accuracy: 4.2 -> 21.0 | Helped |
| add a demonstration to LLaMA-3-70B with Search Only (ablation) | CiteAgent on CiteME · CiteME | accuracy: 21.0 -> 2.5 | Hurt |
| add a demonstration to Claude 3 Opus with Search Only (ablation) | CiteAgent on CiteME · CiteME | accuracy: 26.1 -> 27.7 | Helped |
| add Search and Read commands to Claude 3 Opus (ablation) | CiteAgent on CiteME · CiteME | accuracy: 26.1 -> 27.7 (w/o Demo); 27.7 -> 26.1 (w/ Demo) | Mixed |
| add Search and Read commands to Claude-3.5-Sonnet (ablation) | CiteAgent on CiteME · CiteME | accuracy: 36.1 -> 37.0 (w/o Demo); 43.7 -> 40.3 (w/ Demo) | Mixed |
| add a demonstration to Claude-3.5-Sonnet with Search Only (ablation) | CiteAgent on CiteME · CiteME | accuracy: 36.1 -> 43.7 | Helped |
| add Search and Read commands and a demonstration to o1-preview (ablation) | CiteAgent on CiteME · CiteME | accuracy: 38.7 -> 61.3 | Helped |
| use o1-preview without commands on the benchmark (ablation) | o1-preview on CiteME · CiteME | accuracy: 38.7%; 61.3% | Mixed |
Surprising findings
- Claude 3 Opus and LLaMA-3-70B perform worse with demonstration trajectories in the prompt.
- o1-preview scores 38.7% with no commands, which the authors suggest may reflect memorization of the dataset.
Limitations
- Benchmark has only 130 samples, which limits statistical power.
- Filtering removes items GPT-4o answers correctly, so the benchmark is biased toward items that are hard for GPT-4o and results for other models may be affected by memorization.
- Agent errors include misunderstanding excerpts and stopping prematurely, and 5.8% of runs had technical issues.
Experimental setup
Datasets: FullTextPeerRead ACL-200 RefSeer arXiv citation datasets
Benchmarks: CiteME
Metrics: accuracy error type analysis
Baselines: SPECTER SPECTER2 Claude 3 Opus LLaMA-3-70B Claude 3.5 Sonnet o1-mini o1-preview
Builds on: ReAct SWE-Agent SPECTER SPECTER2
Closest papers by meaning
- HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction (NeurIPS 2024 · similarity 0.85)
- PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies (ICLR 2026 · similarity 0.84)
- Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts (AAAI 2026 · similarity 0.84)
- DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models (NeurIPS 2025 · similarity 0.84)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.83)
- Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.