Research map / Large language models / Retrieval and knowledge
What Is a Good Question? Assessing Question Quality via Meta-Fact Checking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Meta-Fact Checking (MFC), a training-free few-shot prompting method that converts questions into KG reasoning patterns via meta-facts, used both to answer multi-hop KBQA questions and to assess question quality on four knowledge-intensive metrics.
Problem
Automatically assessing knowledge-based question quality from a knowledge-cognition perspective, which requires retrieving the complete (direct and implicit) knowledge underlying a question from a knowledge graph, while existing metrics are linguistic (BLEU, ROUGE) and retrieve-while-reasoning metho
Approach
Decouples KG searching from LLM reasoning by hiding intermediate entities behind a meta-entity ([X]) and passing LLMs natural-language meta-facts summarizing high-level triple types, iterating sub-question reduction until enough facts are gathered, then refining the reasoning path; question quality is scored by matching reasoning path patterns against Bloom-style cognitive levels, unambiguity, dis
Key results
- MFC with GPT-4-turbo reaches 71.1 average EM across five datasets vs 66.3 for ToG and 48.3 for IO prompting.
- On CWQ-QQA, MFC (GPT-4-turbo) achieves F1 of 96.4 on correctness and 76.8 on distractors, outperforming a fine-tuned Llama2-7B on UAM and CRT.
- MFC uses fewer LLM calls than ToG (6.78 vs 11.15 average).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MFC instead of ToG | GPT-3.5-turbo · SimQu, WebQSP, GrailQA, CWQ, and CWQ-QQA | EM accuracy: (+3.3), (+6.7), (+9.8), (+11.0), (+8.3) | Helped |
| Use MFC instead of ToG | GPT-4-turbo · SimQu, GrailQA, CWQ, and CWQ-QQA | EM accuracy: (+1.2), (+5.1), (+6.9), (+13.0) | Helped |
| Use MFC instead of ToG | GPT-4-turbo · WebQSP | EM accuracy: (-2.3) | Hurt |
| Add meta-entity descriptions to the MFC prompt (ablation) | MFC with GPT-3.5-turbo · CWQ | EM accuracy: 54.7 -> 62.8 | Helped |
| Add meta-entity descriptions to the MFC prompt (ablation) | MFC with GPT-3.5-turbo · CWQ | LLM call time: 9.5 -> 8.7 | Helped |
| Add meta-entity descriptions to the MFC prompt (ablation) | MFC with GPT-4-turbo · CWQ | EM accuracy: 68.8 -> 72.2 | Helped |
| Add meta-entity descriptions to the MFC prompt (ablation) | MFC with GPT-4-turbo · CWQ | LLM call time: 9.3 -> 8.4 | Helped |
| Increase search Depth (ablation) | MFC with GPT-3.5-turbo · CWQ | EM accuracy | Helped |
| Set Top-K above 3 (ablation) | MFC with GPT-3.5-turbo · CWQ | EM accuracy | Hurt |
Surprising findings
- KG-enhanced ToG degrades LLM accuracy on Apply and Analyze questions relative to IO and CoT, while MFC does not show this drop as strongly.
Limitations
- Metrics only cover knowledge-based questions, excluding questionnaires and open knowledge questions.
- Cognitive level metric excludes Evaluate and Create levels.
- CWQ-QQA is small (265 examples) and annotated manually.
Experimental setup
Datasets: WebQSP CWQ SimpleQuestions GrailQA Freebase
Benchmarks: SimpleQuestions WebQSP GrailQA CWQ CWQ-QQA
Metrics: exact match accuracy micro-F1 LLM call time
Baselines: IO prompting CoT prompting ToG DiFaR Pangu DECAF Llama2-7B fine-tuned
Builds on: Chain-of-Thought prompting Think-on-Graph
Closest papers by meaning
- Explore What LLM Does Not Know in Complex Question Answering (AAAI 2025 · similarity 0.87)
- Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking (NeurIPS 2025 · similarity 0.85)
- ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models (NeurIPS 2026 · similarity 0.85)
- CK12: A Rounded K12 Knowledge Graph Based Benchmark for Chinese Holistic Cognition Evaluation (AAAI 2024 · similarity 0.85)
- LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI (AAAI 2025 · similarity 0.85)
- MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.