Research map / Large language models / Retrieval and knowledge
RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RepLiQA, a human-written synthetic reading-comprehension and topic-retrieval benchmark whose documents are absent from the web, released in staggered splits to limit contamination, and benchmarks 18 LLMs on it.
Problem
Public QA benchmarks may have leaked into LLM training data, so high scores can reflect memorization rather than reading comprehension on new context.
Approach
Create a benchmark of human-written imaginary reference documents with questions (some unanswerable) and answers that never existed online, and release its five splits on a delayed schedule so later splits remain unseen by models; compare context vs no-context performance against TriviaQA to separate reading from memorization.
Key results
- Models perform significantly better on TriviaQA than on RepLiQA; e.g., Command R+ recall is roughly halved on RepLiQA relative to TriviaQA.
- Mistral Large achieves the highest recall on RepLiQA question answering (72.29%) among the evaluated models, while GPT-4o leads on F1 (61.90) and EM (25.78).
- Larger models do not consistently improve reading-based performance on RepLiQA; GPT-3.5 is likely to outperform GPT-4 in the context-given setting, and Claude Haiku/Command R small variants outperform their larger siblings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate models on RepLiQA's unseen synthetic reference content rather than TriviaQA's factual content | the evaluated LLMs · RepLiQA_0 versus TriviaQA | Recall | Hurt |
| remove the reference document from the prompt (ablation) | the evaluated models · RepLiQA_0 answerable questions | Recall: Mistral Large: 72.29 -> 26.54; GPT-4o: 60.85 -> 8.59; GPT-3.5 Turbo: 58.98 -> 34.67 | Hurt |
| remove the reference document from the prompt (ablation) | the evaluated models · TriviaQA | Recall: Mistral Large: 86.79 -> 88.09; GPT-4o: 86.80 -> 92.96; GPT-3.5 Turbo: 80.98 -> 82.85 | No clear effect |
| increase model size | LLMs evaluated on TriviaQA · TriviaQA | Recall | Helped |
| increase model size | LLMs evaluated on RepLiQA · RepLiQA_0 | Recall | Mixed |
| provide contextual information | the evaluated models · RepLiQA_0 unanswerable questions | unanswerable detection rate | Helped |
Surprising findings
- Without context, GPT-3.5 Turbo's recall on TriviaQA is high, while on RepLiQA models rarely answer correctly without the document, yet models still often produce answers rather than abstaining on unanswerable questions when context is present.
- Gemini Flash 1.5 is top in topic retrieval but sits in the bottom half for question answering, showing task-dependent rankings.
Limitations
- Closed-source models were evaluated via third-party OpenRouter API, so results are hard to reproduce as models change over time.
- Possible bias toward Indian English due to annotator demographics.
- Split 0 was exposed to LLM providers in May 2024, so it is already potentially leaked; some documents were used in APIs as early as January 2024.
Experimental setup
Datasets: TriviaQA
Benchmarks: RepLiQA TriviaQA
Metrics: EM precision recall F1 Rouge-L Meteor BScore F1 for topic retrieval
Baselines: TriviaQA
Builds on: TriviaQA
Closest papers by meaning
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.91)
- SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.90)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.90)
- NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens (ICLR 2025 · similarity 0.90)
- FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering (AAAI 2024 · similarity 0.89)
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.