Research map / Large language models / Retrieval and knowledge
CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CofCA, a step-wise counterfactual multi-hop QA benchmark built by rewriting Wikipedia passages, to evaluate LLM sub-question answering and reasoning chains while reducing data contamination.
Problem
LLMs may answer multi-hop QA from parametric memory rather than by retrieving and integrating evidence, and existing Wikipedia-based MHQA benchmarks are contaminated and only score final answers, not reasoning chains.
Approach
Create counterfactual passages by replacing key entities/noun phrases in Wikipedia passages and back-translating, annotate 2/3/4-hop questions with sub-questions and sub-answers, and evaluate both final answers and full reasoning chains against factual control data.
Key results
- GPT-4 scores 69.9 EM / 82.3 F1 on Wikipedia-based HotpotQA but 53.1 EM / 62.8 F1 on the CofCA 2-hop split.
- GPT-4 achieves only 36.3% fully correct reasoning chains on the 2-hop dataset, while GPT-3.5 gets 13.3%.
- Adding sub-questions to the prompt improves GPT-4 2-hop EM from 43.8 to 53.2 relative to no sub-questions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use counterfactual CofCA questions instead of Wikipedia-based factual questions | GPT-4 · Multi-hop QA; factual datasets compared with CofCA 2-hop | EM: 69.9 -> 53.1 | Hurt |
| use counterfactual CofCA questions instead of Wikipedia-based factual questions | GPT-4 · Multi-hop QA; factual datasets compared with CofCA 2-hop | F1: 82.3 -> 62.8 | Hurt |
| add sub-question information to the prompt (ablation) | GPT-4 · CofCA 2-hop | EM: 43.8 -> 53.2 | Helped |
| add sub-question information to the prompt (ablation) | GPT-4 · CofCA 2-hop | F1: 65.2 -> 67.7 | Helped |
| add sub-question information to the prompt (ablation) | GPT-4 · CofCA 3-hop | EM: 41.4 -> 44.5 | Helped |
| add sub-question information to the prompt (ablation) | GPT-4 · CofCA 3-hop | F1: 61.6 -> 64.5 | Helped |
| add sub-question information to the prompt (ablation) | GPT-4 · CofCA 4-hop | EM: 38.1 -> 42.1 | Helped |
| add sub-question information to the prompt (ablation) | GPT-4 · CofCA 4-hop | F1: 48.9 -> 53.1 | Helped |
| replace original passages with stage 1 keyword-replaced passages (ablation) | GPT-4 · HotpotQA 2-hop questions | EM: 71.35 -> 65.62 | Hurt |
| replace original passages with stage 1 keyword-replaced passages (ablation) | GPT-4 · HotpotQA 2-hop questions | F1: 78.62 -> 70.47 | Hurt |
| paraphrase the stage 1 passages in a second rewriting stage (ablation) | GPT-4 · HotpotQA 2-hop questions | EM: 65.62 -> 63.49 | Hurt |
| paraphrase the stage 1 passages in a second rewriting stage (ablation) | GPT-4 · HotpotQA 2-hop questions | F1: 70.47 -> 67.32 | Hurt |
Surprising findings
- Roughly 69% of GPT-3.5 2-hop cases are total failures on the sub-question chain despite relatively high final QA scores, indicating inflated final-answer performance.
- LLMs sometimes reach correct final answers through incorrect sub-answer chains (about 4 of 20 sampled correct GPT-4 answers).
Limitations
- Dataset size is small (300 passages, 900 counterfactual multi-hop questions), which the authors acknowledge.
- Exact Match underestimates performance due to answer-format variation, so PM via GPT-4-turbo is used as a supplement.
- Counterfactual data was generated with GPT-4 and checked by humans, and only 2-hop passages were used for the annotation-stage ablation.
Experimental setup
Datasets: HotpotQA 2WikiMultihopQA MuSiQue English Wikipedia passages (lucadiliello/english_wikipedia)
Benchmarks: CofCA HotpotQA 2WikiMultihopQA MuSiQue
Metrics: EM F1 partial match (PM) joint EM RC joint F1 RC categorical reasoning-chain EM
Baselines: GPT-4 GPT-3.5 GEMINI-pro text-davinci-003 Bing Chat O1-preview Llama 2-7b Mistral-7b Qwen 2-7b
Builds on: HotpotQA 2WikiMultihopQA MuSiQue chain-of-thought
Closest papers by meaning
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.89)
- Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution (ICML 2026 · similarity 0.88)
- Reasoning Elicitation in Language Models via Counterfactual Feedback (ICLR 2025 · similarity 0.88)
- NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens (ICLR 2025 · similarity 0.88)
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.88)
- MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.