Research map / Large language models / Retrieval and knowledge
From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question-Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills an LM's topical knowledge into a concise "microtheory" of NL sentences, built from GPT-4 fact pools on ARC and MedQA, and uses textual entailment to show it improves grounded QA when added to Wikipedia.
Problem
Chain-of-thought and explanation methods reveal only snippets of an LM's knowledge for single questions, not its overall understanding of a topic, which limits user trust.
Approach
Prompt an LM to generate generic supporting facts for each training question, condense the pool via deduplication and entailment, then select a budget-sized core set (usage, question-coverage, or partial-coverage ILP) that best covers entailment proofs of training hypotheses; the result is a topical microtheory usable with textual entailment.
Key results
- Adding the full fact pool to Wikipedia raises ARC grounding from 84% to 92% of hypotheses and QA accuracy from 69.2% to 78%.
- Expert medical annotators rate partial-coverage microtheory facts 0.6-1.2 points higher than random facts on a 5-point rubric, never dropping below 4.
- Per-question relevance for 1000-fact PC microtheories is about 90% on ARC versus about 70% on MedQA; about 940 training questions are predicted to reach 90% p-relevance on ARC versus 8953 on MedQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the full fact pool as a microtheory to Wikipedia | ARC answer grounding · ARC test topics | hypothesis grounding rate: 84% -> 92% (+8%) | Helped |
| Add a 1000-fact microtheory to Wikipedia | ARC question answering · ARC test topics | QA accuracy: 69.2% -> 73% (+4 points) | Helped |
| Add the full fact pool as a microtheory to Wikipedia | ARC question answering · ARC test topics | QA accuracy: 69.2% -> 78% (+5 points) | Helped |
| Increase microtheory size (ablation) | ARC answer grounding · ARC test topics | hypothesis grounding rate | Helped |
| Increase microtheory size (ablation) | MedQA answer grounding · MedQA test topics | overall grounding rate | No clear effect |
| Use partial coverage optimization rather than question coverage or usage optimization (ablation) | Microtheory training hypothesis coverage · ARC and MedQA training topics | fractional coverage | Helped |
| Select facts randomly rather than using a distilled microtheory (ablation) | ARC question answering · ARC test topics | QA accuracy | Hurt |
| Use a 100-fact random sample instead of the distilled methods (ablation) | ARC answer grounding · ARC test topics | hypothesis grounding rate | Hurt |
| Use any 1000-fact distillation method rather than Wikipedia alone (ablation) | ARC question answering · ARC test topics | QA accuracy: 69.2% -> 73% (+4 points) | Helped |
| Switch among usage, question coverage, and partial coverage optimization (ablation) | 1000-fact microtheories · ARC test topics | QA accuracy | No clear effect |
| Use distilled microtheories rather than random fact selection (ablation) | Microtheory fact relevance · MedQA | expert relevance rating: at least half a grade point | Helped |
| Use MtPC rather than random selection (ablation) | Microtheory fact relevance · MedQA | expert relevance rating: 0.6-1.2 higher | Helped |
Surprising findings
- On MedQA, adding microtheories did not consistently improve overall grounding rates, unlike ARC.
- Over 5000 of the roughly 8000 MedQA facts (in the pool) are never used in an entailment proof, showing most extracted facts are idiosyncratic.
Limitations
- MedQA microtheories give weaker and less consistent grounding gains than ARC, suggesting many idiosyncratic facts in medicine.
- Scaling to large training sets is expensive, requiring many LLM calls per question.
- Relevance judgments are partly done by GPT-4o and by two medical students, so they are approximate.
Experimental setup
Datasets: ARC MedQA EntailmentBank Wikipedia (2021-01-20) medical textbooks
Benchmarks: ARC MedQA
Metrics: grounding rate QA accuracy expert relevance score (1-5) per-question relevance rate p-relevance fact usage rate
Baselines: random fact sample full fact pool WorldTree Wikipedia corpus only
Builds on: TreeWise EntailmentBank chain-of-thought Sentence-BERT IRGR
Closest papers by meaning
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.88)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.88)
- KScope: A Framework for Characterizing the Knowledge Status of Language Models (NeurIPS 2025 · similarity 0.87)
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.87)
- ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models (NeurIPS 2026 · similarity 0.87)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.