Research map / Large language models / Reasoning and chain of thought
KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KAM-CoT, a two-stage multimodal chain-of-thought framework that fuses text, image (DETR/CLIP/caption) and ConceptNet knowledge-graph features via a GNN and cross-attention on T5-Base/FLAN-T5-Base, reaching 93.87% on ScienceQA with ~280M parameters.
Problem
Multimodal chain-of-thought reasoning with small models tends to hallucinate plausible but incorrect rationales and answers; the paper targets knowledge-grounded reasoning on ScienceQA (and A-OKVQA) under a small parameter budget.
Approach
Extends the two-stage MM-CoT pipeline (rationale generation then answer inference) by adding a ConceptNet subgraph encoded with a relational graph attention plus graph convolution network, cross-attended with language and image features and combined by gated fusion (Fusion-1).
Key results
- KAM-CoT FLAN-T5-Base achieves 93.87% average accuracy on ScienceQA test, versus 83.99% for GPT-4 w/ CoT and 75.15% for GPT-3.5 w/ CoT, with 250M-280M parameters.
- Adding the knowledge graph to the captioned T5-Base-style ablation raises accuracy from 87.74% to 92.62% with FLAN-T5-Base (ablation table).
- On A-OKVQA validation, KAM-CoT FLAN-T5-Base reaches 59.65% vs 55.98% for MM-CoT, a 3.67-point gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add knowledge graph grounding (ConceptNet subgraph encoder) to image+caption-context model (ablation) | FLAN-T5-Base multimodal CoT · ScienceQA test | Avg accuracy: 87.74 -> 92.62 | Helped |
| add image features to text-only CoT model (ablation) | FLAN-T5-Base · ScienceQA test | Avg accuracy: 83.42 -> 85.85 | Helped |
| add captions as context (ablation) | image-augmented FLAN-T5-Base · ScienceQA test | Avg accuracy: 85.85 -> 87.74 | Helped |
| use captions for KG node extraction in addition to context (ablation) | T5-Base KAM-CoT · ScienceQA val | Avg accuracy: 92.45 -> 92.48 (captions as context vs context+nodes) | Helped |
| use DETR image encoder instead of CLIP (ablation) | T5-Base KAM-CoT · ScienceQA val | Avg accuracy: 91.02 -> 91.65 | Helped |
| increase number of graph nodes from 50 to 200 (ablation) | T5-Base KAM-CoT · ScienceQA val | Avg accuracy: 88.66 -> 89.51 | Helped |
| gated three-way fusion (Fusion-1) instead of one-step weighted sum (Fusion-3) (ablation) | T5-Base KAM-CoT · ScienceQA val | Avg accuracy: 90.14 -> 91.65 | Helped |
| KAM-CoT (FLAN-T5-Base) vs MM-CoT on ScienceQA test | FLAN-T5-Base · ScienceQA test | Avg accuracy: 87.74 (MM-CoT with captions) -> 93.87 | Helped |
| KAM-CoT vs MM-CoT on A-OKVQA validation | FLAN-T5-Base · A-OKVQA val | Val accuracy: 55.98 -> 59.65 | Helped |
| KAM-CoT graph grounding raises RougeL on ScienceQA (ablation) | FLAN-T5-Base · ScienceQA test | RougeL: 97.27 -> 98.34 | Helped |
Surprising findings
- KAM-CoT with about 35% of training data reportedly outperforms fully trained MM-CoT (FLAN-T5-Base) on ScienceQA (93.87% vs 85.85%).
Limitations
- Parameter count inconsistently reported (254M/279M/280M) and comparison to LLMs uses scores from prior papers rather than matched evaluation.
- Underperforms the best A-OKVQA method (Prophet, 175B) and only evaluated on a multiple-choice variant of A-OKVQA with models under 1B parameters.
- Limited exploration of node count beyond 200 and reliance on ConceptNet triples.
Experimental setup
Datasets: ScienceQA ConceptNet A-OKVQA
Benchmarks: ScienceQA A-OKVQA
Metrics: average accuracy RougeL
Baselines: ViLT Patch-TRM VisualBERT UnifiedQA MM-CoT MM-T-SciQ LLaMA-Adapter LaVIN-13B GPT-3.5 w/ CoT GPT-4 w/ CoT LLaVA (GPT-4) LXMERT
Builds on: MM-CoT QA-GNN GreaseLM Chain-of-thought prompting Relational Graph Attention gated fusion
Closest papers by meaning
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (ICML 2025 · similarity 0.93)
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (NeurIPS 2024 · similarity 0.91)
- CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs (ICLR 2026 · similarity 0.91)
- Uni-CoT: Towards Unified Chain-of-Thought Reasoning Across Text and Vision (ICLR 2026 · similarity 0.90)
- T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Large Language Model Signals for Science Question Answering (AAAI 2024 · similarity 0.90)
- Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.