Research map / Vision-language and multimodal / Visual question answering
SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SK-VQA, a 2M-pair synthetic KB-VQA dataset where GPT-4o generates context documents and QA pairs for images from LAION, Wikipedia (WIT), and COCO-Counterfactuals, and shows that fine-tuning MLLMs on it improves out-of-domain generalization.
Problem
Existing KB-VQA datasets for training MLLMs in context-augmented generation are limited to Wikipedia-linkable images, focus on entity knowledge, and lack question diversity, leading to poor generalization.
Approach
Generate context documents and multiple QA pairs jointly in a single GPT-4o call for images from diverse sources (including synthetic counterfactual images), rather than template-based QA over Wikipedia-linked images, and add image-reference and answer-presence filtering subsets.
Key results
- Zero-shot, LLaVA-v1.6-34B scores 50.02 on SK-VQA, the best among tested models, while Idefics2-8B leads InfoSeek (44.33) and LLaVA-v1.6-34B leads Enc-VQA (77.73).
- Fine-tuning LLaVA-7B on SK-VQA (COCO-CF images with GPT-4 contexts) yields 64.22 average across InfoSeek, Enc-VQA, and ViQuAE, versus 62.99 for LAION-based and 59.85 for Wiki-GPT4-based subsets.
- Human evaluation: 77% annotator accuracy on the full sampled set; 86% of 50 checked QA pairs verified as factual.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune PaliGemma-3B on SK-VQA | PaliGemma-3B · InfoSeek | exact match: 25.66 -> 42.50 | Helped |
| Fine-tune LLaVA-7B on SK-VQA | LLaVA-7B · Enc-VQA | semantic score: 53.69 -> 63.89 | Helped |
| Fine-tune LLaVA-1.6-13B on SK-VQA | LLaVA-1.6-13B · ViQuAE | exact match: 83.75 -> 82.04 | No clear effect |
| Use GPT-4 contexts with COCO-CFs images rather than Wikipedia images and Wikipedia contexts (ablation) | LLaVA-v1.5-7B training · ViQuAE | exact match: 77.95 -> 79.23 | No clear effect |
| Use Wikipedia images with GPT-4 contexts rather than LAION images with GPT-4 contexts (ablation) | LLaVA-v1.5-7B training · InfoSeek | exact match: 44.32 -> 47.00 | Helped |
| Use Wikipedia images with GPT-4 contexts rather than LAION images with GPT-4 contexts (ablation) | LLaVA-v1.5-7B training · Enc-VQA | semantic score: 65.44 -> 53.98 | Hurt |
| Use Wikipedia contexts instead of GPT-4 contexts for Wikipedia images (ablation) | LLaVA-v1.5-7B training · Enc-VQA | semantic score: 53.98 -> 66.67 | Helped |
| Use COCO-CFs images with GPT-4 contexts instead of LAION images with GPT-4 contexts (ablation) | LLaVA-v1.5-7B training · InfoSeek | exact match: 44.32 -> 48.00 | Helped |
| Apply IR filtering to LAION-source training data (ablation) | PaliGemma-3B trained on SK-VQA · ViQuAE | exact match: 71.12 -> 71.48 | Hurt |
| Apply IR and CAP filtering to LAION-source training data (ablation) | PaliGemma-3B trained on SK-VQA · InfoSeek | exact match: 42.50 -> 43.72 | Helped |
| Apply IR and CAP filtering to LAION-source training data (ablation) | PaliGemma-3B trained on SK-VQA · Enc-VQA | semantic score: 55.53 -> 53.72 | Hurt |
| Replace GPT-4 with LLaVA-34B for synthetic QA generation (ablation) | SK-VQA generation · Human-judged generated questions | invalid question rate: 76% | Hurt |
Surprising findings
- Synthetic LLaVA-34B-generated data was judged invalid for 76% of questions, many answerable from context alone, indicating that generator quality matters greatly.
- An LLaMA-3-70b question-only baseline achieves only 9.92% exact match on SK-VQA, suggesting questions require image and context.
Limitations
- Synthetic data generated by a single proprietary model (GPT-4o) may contain factual errors or biases; full accuracy cannot be validated at scale.
- Dataset is limited to English, and some questions can be answered from context alone or from the image alone.
- Factuality checks were limited to 50 QA pairs and automated fact-checking (SAFE) was ineffective for multimodal content.
Experimental setup
Datasets: SK-VQA InfoSeek Encyclopedic-VQA ViQuAE LAION WIT COCO-Counterfactuals
Benchmarks: SK-VQA InfoSeek Encyclopedic-VQA ViQuAE
Metrics: exact match accuracy BEM semantic accuracy human accuracy
Baselines: InfoSeek-trained model Enc-VQA-trained model PaliGemma-3B LLaVA-v1.5-7B LLaVA-v1.6-7B Qwen-VL-7B Idefics2-8B LLaVA-v1.6-34B
Builds on: LLaVA InfoSeek Encyclopedic-VQA BERTopic COCO-Counterfactuals
Closest papers by meaning
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.91)
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · similarity 0.91)
- Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines (AAAI 2025 · similarity 0.91)
- Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering (NeurIPS 2025 · similarity 0.90)
- SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge (NeurIPS 2024 · similarity 0.90)
- BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.