Research map / Large language models / Retrieval and knowledge

The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Knowledge editing in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the FACTS Leaderboard, a Kaggle-hosted suite of four factuality benchmarks (Multimodal, Parametric, Search, Grounding v2) aggregated into a single FACTS Score, and evaluates 15 proprietary LLMs on it.

Problem

Existing factuality benchmarks each cover a narrow capability (grounding to context, parametric recall, or search use), so there is no holistic measure of LLM factual reliability across modalities, knowledge sources and response formats.

Approach

Aggregate four sub-leaderboards covering multimodal grounded QA, closed-book parametric factoid QA (adversarially filtered with open-weight models), search-tool QA, and updated long-form grounding with new judges and eligibility checks into one FACTS Score, with partly private test sets hosted on Kaggle.

Key results

• Gemini 3 Pro achieves the top FACTS Score of 68.8, with the highest Parametric (76.4) and Search (83.8) accuracy among 15 evaluated models.
• On FACTS Multimodal the best accuracy is about 46.9% (Gemini 2.5 Pro), and Grok 4 Fast scores lowest at 17.7%.
• Multimodal autorater reaches macro F1 of 72.3 (Coverage) and 78.2 (No-Contradiction) against human annotations; grounding judge selection via Macro-F1 chose Gemini 2.5 Flash and GPT-5 with v2 prompt (65.33, 65.18).

Surprising findings

• Claude models show high attempted accuracy but low overall accuracy on Parametric because they hedge heavily (e.g., Claude 4 Sonnet hedges 45.1%).
• Gemini 3 Pro performs fewer searches on average than other top models, while Grok models search the most.

Limitations

• Only a subset of prompts is public; full evaluation is run by Kaggle, so results depend on the private set.
• Does not cover video understanding, fast-changing information, or knowledge-base tool-use factuality, as acknowledged by the authors.
• Autorater agreement is moderate (Spearman 0.64 for Coverage; macro F1 72.3 and 78.2).

Experimental setup

Datasets: FACTS Grounding v1 prompts Wikipedia Google Knowledge Graph (KG Hops, Wiki Two-Hop) Wikimedia/iNaturalist/DOCCI images

Benchmarks: FACTS Multimodal FACTS Parametric FACTS Search FACTS Grounding v2

Metrics: accuracy F1 attempted accuracy hedging rate coverage no-contradiction macro-F1 FACTS Score

Baselines: Gemini 2.5 Pro GPT-5 Grok 4 o3 Claude 4.5 Opus

Builds on: FACTS Grounding SimpleQA

Closest papers by meaning

• Long-form factuality in large language models (NeurIPS 2024 · similarity 0.91)
• Fine-Tuning Language Models for Factuality (ICLR 2024 · similarity 0.89)
• ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models (NeurIPS 2026 · similarity 0.88)
• Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation (NeurIPS 2025 · similarity 0.87)
• FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees (ICML 2025 · similarity 0.86)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.86)