Research map / Large language models / Retrieval and knowledge
Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Chain-of-Knowledge (CoK), a three-stage framework that progressively corrects LLM rationales using heterogeneous knowledge sources (Wikidata, tables, text corpora) with an adaptive query generator producing SPARQL, SQL, or natural-language queries.
Problem
LLMs hallucinate on knowledge-intensive questions; existing retrieval-augmented chain-of-thought methods use a fixed knowledge source, struggle with structured queries, and propagate errors across rationale steps.
Approach
Self-consistency identifies uncertain questions; for those, knowledge domains are selected, and each rationale is sequentially corrected with retrieved knowledge from domain-matched sources (via an adaptive query generator choosing query language per source), so later rationales build on corrected earlier ones before answer consolidation.
Key results
- CoK (6-shot) reaches 73.3% on MedMCQA vs 66.4% for CoT, and averages about 4.3% improvement over CoT across tasks.
- On HotpotQA (3-shot), CoK reaches 34.1% EM versus 29.9% for CoT and 31.8% for Verify-and-Edit; parallel correction gives only 31.2%.
- In a human study, CoK rationales were judged more factually consistent than CoT-SC rationales (56% CoK vs 14% CoT-SC across all predictions).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CoK instead of CoT | gpt-3.5-turbo-0613 · FEVER (3-shot; 6-shot) | accuracy: 57.8% -> 63.4%; 55.6% -> 58.5% | Helped |
| use CoK instead of CoT | gpt-3.5-turbo-0613 · HotpotQA (3-shot; 6-shot) | exact match: 29.9% -> 34.1%; 34.4% -> 35.4% | Helped |
| use CoK instead of CoT | gpt-3.5-turbo-0613 · FeTaQA (3-shot; 6-shot) | BLEU: 17.3 -> 25.0; 19.4 -> 26.0 | Helped |
| use CoK instead of CoT | gpt-3.5-turbo-0613 · MedMCQA (3-shot; 6-shot) | accuracy: 59.6% -> 70.5%; 66.4% -> 73.3% | Helped |
| use CoK instead of CoT | gpt-3.5-turbo-0613 · MMLU Physics (3-shot; 6-shot) | accuracy: 41.9% -> 45.5%; 43.5% -> 47.0% | Helped |
| use CoK instead of CoT | gpt-3.5-turbo-0613 · MMLU Biology (3-shot; 6-shot) | accuracy: 81.5% -> 83.0%; 81.7% -> 84.4% | Helped |
| replace parallel editing with progressive dynamic knowledge adapting (ablation) | CoK · HotpotQA (3-shot) | exact match: 31.2% -> 34.1% | Helped |
| add UpToDate to Flashcard (ablation) | CoK on MedMCQA · MedMCQA (3-shot) | accuracy: 67.1% -> 69.2% | Helped |
| add Biology-domain sources to Medical-domain sources (ablation) | CoK on MedMCQA · MedMCQA (3-shot) | accuracy: 69.2% -> 70.5% | Helped |
Surprising findings
- CoT underperforms standard prompting on FeTaQA, MedMCQA, and MMLU Physics, indicating hallucinated rationales hurt knowledge-intensive tasks.
- ChatGPT achieves only 8.9% on SPARQL generation evaluation, whereas the fine-tuned LLaMA-2-7B reaches 41.1%.
Limitations
- Relies on external knowledge sources that may be unreliable or conflicting.
- Retrieval may return noisy or irrelevant results when queries are inaccurate.
- Failure cases can stem from LLM reasoning errors even after factual correction.
Experimental setup
Datasets: LC-quad KQA-pro Medical Flashcards ScienceQA WikiSQL Wikidata Wikipedia UpToDate PhysicsClassroom CK-12
Benchmarks: FEVER HotpotQA FeTaQA MedMCQA MMLU Physics MMLU Biology
Metrics: accuracy exact match BLEU factual accuracy (ProgramFC) human preference
Baselines: Standard prompting CoT CoT-SC Verify-and-Edit ReAct
Builds on: chain-of-thought prompting self-consistency Verify-and-Edit ReAct chain-of-hindsight
Closest papers by meaning
- Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding (ICLR 2024 · similarity 0.88)
- Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models (ICML 2024 · similarity 0.88)
- CogSQL: A Cognitive Framework for Enhancing Large Language Models in Text-to-SQL Translation (AAAI 2025 · similarity 0.88)
- Metastable Dynamics of Chain-of-Thought Reasoning: Provable Benefits of Search, RL and Distillation (ICML 2025 · similarity 0.88)
- Markovian Transformers for Informative Language Modeling (ICLR 2026 · similarity 0.87)
- OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.