Research map / Large language models / Agents, code and math
CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CURIE, a benchmark of 10 expert-curated long-context scientific tasks (580 examples from 429 papers across six disciplines), and evaluates eight LLMs, finding best scores around 32% and proposing LLMSim and LMScore model-based metrics.
Problem
Existing LLM benchmarks in science focus on short inputs and multiple-choice questions, so they do not measure long-context scientific comprehension, reasoning and information extraction from full papers.
Approach
A multitask benchmark built from full-length scientific papers and expert annotations across materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity and proteins, paired with model-based metrics (LLMSim for structured outputs, LMScore for free text) for evaluating mixed-format responses.
Key results
- Claude 3 Opus is the best-performing model on average across the 10 tasks, with consistent high performance; the best overall performance is reported at about 32%.
- GPT-4o scores poorly on PDB (ID_r 0.13) due to repetition, while Gemini 2.0 Flash reaches ID_r 0.56 on PDB.
- LLMSim precision and recall were reported to closely match human expert measurements on Gemini 1.5 Pro and GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gemini 2.0 Flash rather than GPT-4o | CURIE PDB sequence reconstruction · PDB | Identity ratio: 0.13 -> 0.56 | Helped |
| Use Gemini 2.0 Flash rather than GPT-4o | CURIE GEO extraction · GEO | ROUGE-L: 27.30 -> 29.69 | Helped |
| Use Gemini 2.0 Flash rather than GPT-4o | CURIE BIOGR georeferencing · BIOGR | IoU: 0.37 -> 0.49 | Helped |
| Use Gemini 1.5 Pro rather than GPT-4o | CURIE HFD derivation · HFD | ROUGE-L: 9.81 -> 18.86 | Helped |
| Use Gemini 2.0 Flash rather than GPT-4o | CURIE HFE equation extraction · HFE | ROUGE-L: 48.93 -> 50.06 | Helped |
| Use Gemini 2.0 Flash rather than GPT-4o | CURIE DFT tasks · DFT | ROUGE-L: 13.30 -> 14.18 | Helped |
| Use Gemini 1.5 Pro rather than GPT-4o | CURIE MPV extraction · MPV | ROUGE-L: 12.74 -> 31.54 | Helped |
| Use Gemini 1.5 Flash rather than GPT-4o | CURIE QECC entry generation · QECC | ROUGE-L: 20.02 -> 23.50 | Helped |
| Restrict MPV extraction to non-trivial properties (ablation) | CURIE MPV extraction · MPV non-trivial properties | F1: 22.82 -> 38.32 | Mixed |
| Restrict MPV extraction to refractive index and optical bandgap (ablation) | CURIE MPV extraction · MPV specific properties | F1: 34.04 -> 31.48 | Hurt |
Surprising findings
- GPT-4o, a popular strong model, fails dramatically on the protein sequence reconstruction task due to repetition and failure to stop generation.
- Mixtral outperforms GPT-4o on the PDB task.
Limitations
- Limited scale: only 580 examples across a narrow set of domains and tasks.
- Rouge-L and BERTScore can be unforgiving for reasonable but differently formatted responses; model-based metrics are far from perfect.
- Human performance is hard to obtain since no single person has expertise across all domains.
Experimental setup
Datasets: CURIE
Benchmarks: CURIE
Metrics: ROUGE-L BERTScore F1 LLMSim F1 IoU identity ratio LMScore
Baselines: GPT-4o Claude 3 Opus Gemini 1.5 Pro Gemini 1.5 Flash Gemini 1.0 Pro Mixtral Command-R+ LongLLaMA
Builds on: ROUGE-L BERTScore Prometheus-Vision LIMA
Closest papers by meaning
- SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models (NeurIPS 2024 · similarity 0.89)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · similarity 0.88)
- LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models (ICML 2025 · similarity 0.88)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.88)
- SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.