Research map / Vision-language and multimodal / Visual question answering
Multimodal Commonsense Knowledge Distillation for Visual Question Answering (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a graph-based multimodal commonsense knowledge distillation framework where a GCN teacher over commonsense (ATOMIC2020), visual and question nodes distills soft labels to small or large student VQA models, evaluated on ScienceQA.
Problem
Multimodal LLMs and VLPMs struggle on VQA questions requiring external commonsense knowledge, and fine-tuning large models is computationally expensive.
Approach
Build a heterogeneous per-sample graph joining image, question, and retrieved ATOMIC2020 commonsense triplets (via VLPM embedding similarity), train a GCN teacher on it, and distill its soft labels into any student model (MLP, Transformer, ViLT, VisualBERT, UnifiedQA) without fine-tuning the large models.
Key results
- Teacher + MLP reaches 53.92 AVG vs 42.71 for MLP alone on ScienceQA.
- Teacher + UnifiedQA-base reaches 72.33 AVG vs 70.12 for UnifiedQA-base.
- Teacher + ViLT reaches 65.41 AVG vs 61.14 for ViLT alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply graph-based commonsense knowledge distillation to MLP | MLP · ScienceQA | micro F1 (AVG): 42.71 -> 53.92 | Helped |
| Apply graph-based commonsense knowledge distillation to Transformer | Transformer · ScienceQA | micro F1 (AVG): 48.35 -> 56.79 | Helped |
| Apply graph-based commonsense knowledge distillation to ViLT | ViLT · ScienceQA | micro F1 (AVG): 61.14 -> 65.41 | Helped |
| Apply graph-based commonsense knowledge distillation to VisualBERT | VisualBERT · ScienceQA | micro F1 (AVG): 61.87 -> 65.69 | Helped |
| Apply graph-based commonsense knowledge distillation to UnifiedQAbase | UnifiedQAbase · ScienceQA | micro F1 (AVG): 70.12 -> 72.33 | Helped |
Limitations
- Evaluated only on ScienceQA; no other VQA benchmarks reported.
- Gains over large baselines are modest (e.g., VisualBERT 61.87 to 65.69 is the largest but still below UnifiedQA-based results).
- Method description lacks detail on hyperparameters, compute cost, and the exact graph statistics.
Experimental setup
Datasets: ScienceQA ATOMIC2020
Benchmarks: ScienceQA
Metrics: micro F1-score accuracy by subject (NAT, SOC, LAN, AVG)
Baselines: MLP Transformer ViLT VisualBERT UnifiedQA-base
Builds on: VQA-GNN Graph Convolutional Network ATOMIC2020 ViLT VisualBERT UnifiedQA
Closest papers by meaning
- BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining (AAAI 2024 · similarity 0.90)
- Multimodal Graph Networks for Compositional Generalization in Visual Question Answering (NeurIPS 2020 · similarity 0.90)
- Object Attribute Matters in Visual Question Answering (AAAI 2024 · similarity 0.89)
- Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.89)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.89)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.