Research map / Large language models / Retrieval and knowledge
NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NovelQA, a long-context QA benchmark over 89 English novels (2,305 QA pairs, average over 200K tokens), documented with a datasheet.
Problem
Existing benchmarks do not adequately evaluate LLM long-context understanding over very long, complex narratives (documents exceeding 200K tokens).
Approach
A QA benchmark built from 89 full-length novels with human-written questions, gold answers, evidence, and GPT-4-generated complexity/aspect labels and multiple-choice options, presented as a datasheet-documented test set.
Key results
- NovelQA contains 2,305 QA pairs over 89 novels, 24 copyright-protected and 65 public-domain.
- Three rounds of author review over four months; only 79.4% of collected data preserved after cleaning.
- Inter-annotator agreement reported as 94.6%.
Limitations
- Copyright-protected books and gold answers/evidence are withheld, limiting reproducibility.
- Some annotation errors are inevitable despite review.
- Complexity and aspect labels and multiple-choice options are GPT-4 generated, with manual checking only.
Experimental setup
Datasets: NovelQA
Benchmarks: NovelQA
Metrics: accuracy inter-annotator agreement
Builds on: Datasheets for Datasets
Closest papers by meaning
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.90)
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.89)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.89)
- SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.88)
- CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark (ICLR 2025 · similarity 0.88)
- NoLiMa: Long-Context Evaluation Beyond Literal Matching (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.