Research map / Large language models / AI and society
GAIA: a benchmark for General AI Assistants
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GAIA, a benchmark of 466 human-crafted, real-world general assistant questions with unambiguous factoid answers, and shows humans score ~92% while GPT-4 with plugins reaches only ~30% at Level 1 and 0% at Level 3.
Problem
Existing LLM benchmarks are saturated, rely on multiple-choice or closed synthetic environments, and are vulnerable to contamination, making them poor measures of general AI assistant capabilities that need tool use, multi-modality and web browsing.
Approach
Design conceptually simple but multi-step real-world questions with single, factual, easily verifiable answers that are hard to memorize, validated by independent annotators, and evaluated by quasi exact match in zero-shot settings.
Key results
- Human annotators score 92% on GAIA (93.9% Level 1, 91.8% Level 2, 87.3% Level 3).
- GPT-4 with manually selected plugins reaches 30.3% on Level 1 and 9.7% on Level 2, with 0% on Level 3.
- GPT-4 without plugins scores 9.1% on Level 1 and 2.6% on Level 2; a search engine baseline scores 7.4% on Level 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| augment GPT4 with tools/plugins (manually selected) versus plain GPT4 | GPT4 · GAIA Level 1 and Level 2 | score %: Level 1 9.1 to 30.3; Level 2 2.6 to 9.7 | Helped |
| AutoGPT with GPT4 backend (automatic tool selection) versus GPT4 without plugins | AutoGPT · GAIA Level 2 | score %: 0.4 vs 2.6 for GPT4 | Hurt |
| AutoGPT with GPT4 backend versus GPT4 without plugins | AutoGPT · GAIA Level 1 | score %: 14.4 vs 9.1 for GPT4 | Mixed |
| all tested LLM assistants on Level 3 questions | GPT4, GPT4 Turbo, AutoGPT, GPT4 + plugins · GAIA Level 3 | score %: 0 for all | No clear effect |
Surprising findings
- AutoGPT with GPT-4 backend underperforms plain GPT-4 with plugins and even GPT-4 without plugins on Level 2 (0.4%) despite using tools.
- Search engine results alone answer 7.4% of Level 1 questions, showing some questions are findable on the first results page.
Limitations
- Does not evaluate the reasoning trace leading to the answer, only the final answer.
- Covers only English and mostly English web sources, lacking linguistic and cultural diversity.
- Only 166 developer questions are released with annotations; GPT-4 plugin results are not exactly reproducible due to manual plugin selection.
Experimental setup
Benchmarks: GAIA
Metrics: quasi exact match accuracy average time to answer
Baselines: GPT-4 GPT-4 Turbo AutoGPT GPT-4 + plugins Search engine Human annotator
Builds on: AutoGPT ChatGPT plugins
Closest papers by meaning
- AGI-Elo: How Far Are We From Mastering A Task? (NeurIPS 2025 · similarity 0.86)
- GTA: A Benchmark for General Tool Agents (NeurIPS 2024 · similarity 0.86)
- Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments (ICLR 2026 · similarity 0.84)
- GISA: A Benchmark for General Information-Seeking Assistant (NeurIPS 2026 · similarity 0.84)
- \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.84)
- SKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.