Research map / Large language models / Retrieval and knowledge

ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Explanatory and multiple-choice QA benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ListQA, a 9,045-question Wikipedia-table-grounded benchmark for multi-element list-formatted factual recall, with an LLM-judge F1 metric based on optimal assignment, evaluating 34 models.

Problem

Existing factual benchmarks mostly target single answers, so they do not measure whether LLMs can recall complete sets of facts, avoid hallucinated entries, and structure them correctly.

Approach

Frames list-element evaluation as a linear sum assignment problem: ROUGE-weighted optimal one-to-one matching of predicted to reference elements, followed by independent LLM-judge grading of each matched pair, yielding an LLM-Judge-F1 that reduces judge calls from O(n^2) to O(n). The benchmark pairs simple and multi-faceted list questions with evergreen temporal constraints.

Key results

• The best standard-inference model (Gemini 3.1 Pro with thinking is top; best standard is GPT-5.4 at 36.5%) scores below 37% LLM-Judge-F1 on ListQA.
• LLM-Judge-F1 correlates with human F1 at Pearson r = 0.965 over 3,313 human-graded elements.
• Oracle RAG with the source Wikipedia page raises LLM-Judge-F1 by 34-65 points across evaluated models, e.g., Gemma 4 31B from 16.0 to 80.5.

Surprising findings

• Thinking improves factual accuracy universally but its effect on format compliance differs by model family, with closed-source models slightly losing format compliance while open-weight models gain it.
• Many models score higher on the complex subset format than the simple subset because they default to sub-bullet formatting unconditionally.

Limitations

• English-only and US-centric (78% of samples are US entities).
• Does not reward abstention; no explicit measure of knowing limits.
• Reference completeness verified only against a single Wikipedia table; LLM-judge cost and noise.

Experimental setup

Datasets: Wikipedia tables

Benchmarks: ListQA

Metrics: LLM-Judge-F1 Format-Correctness Relaxed-Recall Relaxed-Precision ROUGE-F1 LLM-Judge-F1-Ordered

Baselines: SimpleQA QAMPARI RoMQA FActScore FACTS Grounding

Builds on: SimpleQA grading template ROUGE Jonker-Volgenant algorithm

Closest papers by meaning

• RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.89)
• Long-form factuality in large language models (NeurIPS 2024 · similarity 0.88)
• The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality (NeurIPS 2026 · similarity 0.88)
• CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark (ICLR 2025 · similarity 0.88)
• Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.87)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval (ICLR 2025 · similarity 0.87)