Research map / Large language models / Retrieval and knowledge
PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PhantomWiki, an on-demand pipeline that generates fictional universes of templated Wikipedia-style articles with CFG-generated multi-hop questions and Prolog-verified answers, to evaluate LLM reasoning and retrieval without data leakage.
Problem
Static QA and reasoning benchmarks are prone to data contamination and memorization, and existing perturbation-based or dynamic datasets struggle with factual consistency or focus on real-time knowledge rather than knowledge-independent reasoning and retrieval.
Approach
Generate a fresh, fully synthetic fictional universe per evaluation: family trees and facts are rendered as templated articles, questions come from a context-free grammar with depth-controlled recursion, and answers are computed exactly by a Prolog program, so reasoning difficulty and corpus size can be varied independently.
Key results
- In-context CoT with DeepSeek-R1-32B reaches 52.42% F1 at n=50, and F1 drops for all models as universe size grows to n=5000 (e.g., GPT-4o ReAct 36.85% at n=5000).
- Agentic ReAct prompting avoids the steep RAG decline; at n=5000 Llama-3.3-70B ReAct reaches 30.89% F1 versus 15.07% for zero-shot RAG.
- GRPO fine-tuning of Qwen2.5-3B (LoRA) raises F1 from 16.82% (zero-shot) to 31.38%, while SFT gives no gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use in-context CoT instead of in-context ZeroShot | DeepSeek-R1-32B · PhantomWiki n=50 | F1 (%): 42.42 ± 1.69 -> 52.42 ± 2.64 | Helped |
| Use in-context CoT instead of in-context ZeroShot | GPT-4o · PhantomWiki n=500 | F1 (%): 16.76 ± 0.87 -> 41.02 | Helped |
| Use RAG CoT instead of RAG ZeroShot | GPT-4o · PhantomWiki n=50 | F1 (%): 28.05 ± 2.48 -> 20.49 ± 1.07 | Hurt |
| Use RAG CoT instead of RAG ZeroShot | Gemini-1.5-Flash · PhantomWiki n=50 | F1 (%): 28.92 ± 2.60 -> 20.12 ± 1.69 | Hurt |
| Use ReAct instead of RAG prompting | GPT-4o · PhantomWiki n=5000 | F1 (%): 18.13 ± 0.66 -> 36.85 | Helped |
| Fine-tune with GRPO instead of ZeroShot prompting | Qwen2.5-0.5B · PhantomWiki n=50 | F1 (%): 11.78 ± 0.94 -> 13.25 ± 0.93 | Helped |
| Fine-tune with GRPO instead of ZeroShot prompting | Qwen2.5-3B (LoRA) · PhantomWiki n=50 | F1 (%): 16.82 ± 2.37 -> 31.38 ± 0.86 | Helped |
| Fine-tune with SFT instead of ZeroShot prompting | Qwen2.5-0.5B · PhantomWiki n=50 | F1 (%): 11.78 ± 0.94 -> 11.71 ± 1.10 | No clear effect |
| Fine-tune with SFT instead of ZeroShot prompting | Qwen2.5-3B (LoRA) · PhantomWiki n=50 | F1 (%): 16.82 ± 2.37 -> 16.89 ± 2.22 | No clear effect |
| Use CoT instead of ZeroShot prompting | Qwen2.5-0.5B · PhantomWiki n=50 | F1 (%): 11.78 ± 0.94 -> 2.68 ± 0.22 | Hurt |
| Increase universe size | LLMs across prompting techniques · PhantomWiki | F1 (%) | Hurt |
| Increase the number of reasoning steps | LLMs across prompting techniques · PhantomWiki n=50 | F1 (%) | Hurt |
Surprising findings
- Frontier LLMs struggle on PhantomWiki even though articles are short templated text and many questions are trivially solvable for humans.
- Chain-of-thought does not always help in the RAG setting; GPT-4o and Gemini-1.5-Flash score lower with CoT RAG than zero-shot RAG.
Limitations
- Articles are minimal templated text, not naturalistic; LLM rephrasing was not adopted due to hallucination risk.
- Question types limited to family relations, attributes and aggregation; the grammar is restricted in scope.
- RAG uses only BM25 because neural retrievers are a poor fit for synthetic text.
Experimental setup
Datasets: PhantomWiki synthetic universes
Benchmarks: PhantomWiki
Metrics: answer-level F1 runtime
Baselines: ZeroShot in-context CoT in-context ZeroShot RAG CoT RAG ReAct
Builds on: CLUTRR ReAct GRPO chain-of-thought
Closest papers by meaning
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.88)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.87)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.86)
- WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia (NeurIPS 2024 · similarity 0.86)
- MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.