Research map / Large language models / Retrieval and knowledge
MEQA: A Benchmark for Multi-hop Event-centric Question Answering with Explanations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MEQA, a 2,243-question multi-hop event-centric QA benchmark built semi-automatically from WikiEvents event structures, with QA-format explanations scored by completeness and logical consistency metrics.
Problem
Existing multi-hop QA benchmarks focus on entity-entity reasoning and lack event-centric multi-hop questions requiring reasoning over both entities and events, with faithful explanations.
Approach
Bottom-up question generation that composes event structures from an information extraction dataset (WikiEvents) into event reasoning chains, filters reasoning shortcuts, merges chains, and converts them to synthetic multi-hop questions with QA-style explanations that humans then curate.
Key results
- Human accuracy on MEQA is about 88% versus ChatGPT few-shot F1 of 0.238, showing a large gap.
- Adding Full Event KG to CoT-QA raises GPT-3.5 answer F1 from 0.294 to 0.494 and completeness F1 from 0.200 to 0.421.
- Completeness and logical consistency metrics correlate with human judgments at 0.693 and 0.601.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Entity KG (ablation) | T5 · MEQA | F1: 0.2831 -> 0.2942 | Helped |
| add Full Event KG (ablation) | Fewshot · MEQA | F1: 0.2377 -> 0.4581 | Helped |
| add Entity (ablation) | CoT-QA · MEQA | F1: 0.2940 -> 0.3500 | Helped |
| add Entity KG (ablation) | CoT-QA · MEQA | F1: 0.2940 -> 0.3344 | Helped |
| add Event Triggers (ablation) | CoT-QA · MEQA | F1: 0.2940 -> 0.3120 | Helped |
| add Event Triggers + Arguments (ablation) | CoT-QA · MEQA | F1: 0.2940 -> 0.4471 | Helped |
| add Full Event KG (ablation) | CoT-QA · MEQA | F1: 0.2940 -> 0.4940 | Helped |
| add Full Event KG (ablation) | CoT-Freeform · MEQA | F1: 0.1494 -> 0.3823 | Helped |
| use CoT-Freeform instead of Fewshot | GPT-3.5-turbo-1106 · MEQA | recall: 0.5360 -> 0.5392 | No clear effect |
| use CoT-QA instead of Fewshot | GPT-3.5-turbo-1106 · MEQA | F1: 0.2377 -> 0.2940 | Helped |
Surprising findings
- CoT-QA explanations have the highest precision but lowest answer F1 among the prompting variants, with CoT-Freeform scoring highest on logical consistency mainly due to short outputs.
- Providing structured event graphs helps more than entity lists or event triggers alone, and entity or trigger hints sometimes mislead the model.
Limitations
- Covers only English documents, limiting multilingual evaluation.
- Relies on WikiEvents's 211 documents, a small source corpus.
- Possible data leakage of WikiEvents into LLM pretraining, only partially addressed.
Experimental setup
Datasets: WikiEvents MEQA HotpotQA 2WikiMultihopQA TORQUE
Benchmarks: MEQA HotpotQA 2WikiMultihopQA
Metrics: precision recall F1 completeness logical consistency
Baselines: ChatGPT few-shot CoT-QA CoT-Freeform T5 fine-tuned Claude-3-Haiku Llama3-70B
Builds on: HotpotQA WikiEvents chain-of-thought prompting KAIROS schema
Closest papers by meaning
- Question Calibration and Multi-Hop Modeling for Temporal Question Answering (AAAI 2024 · similarity 0.89)
- Tree-of-Reasoning Question Decomposition for Complex Question Answering with Large Language Models (AAAI 2024 · similarity 0.88)
- Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS 2025 · similarity 0.87)
- CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark (ICLR 2025 · similarity 0.86)
- Autofocus Retrieval: An Effective Pipeline for Multi-Hop Question Answering With Semi-Structured Knowledge (NeurIPS 2026 · similarity 0.86)
- Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answering (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.