Research map / Large language models / Retrieval and knowledge
HiddenPathQA: A Benchmark for Knowledge Graph Question Answering When Questions Hide Their Paths
Sumin Jo, Edward Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HiddenPathQA, a KGQA benchmark whose questions omit lexical cues to the gold knowledge-graph path, so systems must infer multi-hop schema paths from implicit phrasing.
Problem
Existing KGQA evaluations use questions whose wording lexically exposes the gold KG path, whereas real users ask without knowing the schema, so current systems may be overestimated.
Approach
A benchmark where questions compress multi-hop KG paths into single natural phrases (e.g., "family history" for a 4-hop father/medical condition chain), testing path inference rather than lexical matching.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace hop-by-hop pruning with holistic chain selection (TieredHolistic) | KGQA LLM pipeline · HiddenPathQA Pseudoword (GPT-5-mini) | accuracy: 41.7 -> 59.6 | Improved |
| replace hop-by-hop pruning with holistic chain selection (FlatHolistic) | KGQA LLM pipeline · HiddenPathQA Original (Qwen3-32B) | accuracy: 69.1 -> 78.5 | Improved |
| remove entity names (pseudoword replacement) | closed-book CoT / Zero-shot · HiddenPathQA | accuracy: 64.6 -> 25.3 (Qwen3-32B) | Hurt |
| add KG retrieval to LLM (ToG, FiDeLiS, KARPA) on Original split | LLM · HiddenPathQA Original | accuracy: 69.1 -> 44.3 (GPT-5-mini CoT vs KARPA) | Hurt |
| add KG retrieval to LLM on Pseudoword split | LLM · HiddenPathQA Pseudoword | accuracy: 23.1 -> 41.7 (GPT-5-mini CoT vs KAPING) | Improved |
| expose all chains at once (Flat) vs depth-by-depth (Tiered) (ablation) | Holistic Path Selection · HiddenPathQA (Pseudoword, GPT-5-mini) | accuracy: 59.6 (Tiered) vs 51.5 (Flat); Qwen 56.0 vs 56.3 | Mixed |
| holistic path selection vs baselines on path retrieval | Holistic Path Selection · HiddenPathQA | Path F1: 49.2 -> 66.9 (Qwen3-32B, R2-KG vs Flat) | Improved |
Experimental setup
Benchmarks: HiddenPathQA
Closest papers by meaning
- KG-Guard: Graph-Based Hallucination Detection for Knowledge Base Question Answering (NeurIPS 2026 · similarity 0.91)
- Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking (NeurIPS 2025 · similarity 0.89)
- Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models (AAAI 2025 · similarity 0.88)
- EPERM: An Evidence Path Enhanced Reasoning Model for Knowledge Graph Question and Answering (AAAI 2025 · similarity 0.88)
- Autofocus Retrieval: An Effective Pipeline for Multi-Hop Question Answering With Semi-Structured Knowledge (NeurIPS 2026 · similarity 0.88)
- Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
