Research map / Vision-language and multimodal / Multimodal LLMs
VIGC: Visual Instruction Generation and Correction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIGC, a self-instruct framework where a vision-language model (MiniGPT-4+) generates visual instruction-tuning QA data and corrects hallucinations via iterative Q-Former updates, then uses the data to finetune LLaVA and InstructBLIP.
Problem
High-quality multimodal instruction-tuning data is scarce; LLaVA-style pipelines depend on language-only GPT-4 with pre-annotated captions and boxes, losing image detail, while existing MLLMs generating data tend to hallucinate.
Approach
Fine-tune a VLM's Q-Former with instruction templates so it generates diverse image QA pairs (Visual Instruction Generation), then iteratively re-generate answers conditioned on the question and previous answer (Iterative Q-Former, Visual Instruction Correction) to reduce hallucination.
Key results
- Adding 36K VIC-corrected detailed-description and conversation data to LLaVA-7B raised LLaVA-Eval All score from 81.0 to 85.8 (coco), surpassing LLaVA-13B* at 84.8.
- Hallucination in VIG-generated samples fell from 66 to 10 flagged cases on 100 images after VIC correction.
- Fine-tuning InstructBLIP with generated data improved OKVQA from 63.1 to 63.8 and A-OKVQA from 62.5 to 64.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VIGC-generated data corrected by VIC (vs. uncorrected VIG) for detail description (ablation) | LLaVA-7B · LLaVA-Eval | All relative score: 81.4 -> 85.8 | Helped |
| add VIGC data corrected by VIC for conversation (ablation) | LLaVA-7B · LLaVA-Eval | All relative score: 83.3 (VIG) -> 84.0 (VIC) overall; Conv subscore 80.9 -> 83.9 | Helped |
| add VIGC data corrected by VIC for complex reasoning (ablation) | LLaVA-7B · LLaVA-Eval | All relative score: 82.6 (VIG) -> 83.3 (VIC) overall, but Conv subscore drops 81.4 -> 80.2 | Mixed |
| add VIGC-generated detail description data (VIG only, uncorrected) (ablation) | LLaVA-7B · LLaVA-Eval | All relative score: 81.0 -> 81.4 | Helped |
| add VIGC coco-extra data to LLaVA-7B training | LLaVA-7B · LLaVA-Eval | All relative score: 81.0 -> 85.8 (+4.8) | Helped |
| add VIGC Objects365 data to LLaVA-7B training | LLaVA-7B · LLaVA-Eval | All relative score: 81.0 -> 85.2 (+4.2) | Helped |
| add VIGC coco data to LLaVA-13B training | LLaVA-13B · LLaVA-Eval | All relative score: 84.8 -> 86.8 (+2.0) | Helped |
| add VIGC coco data to MiniGPT-4+ training | MiniGPT-4+ · MMBench | Overall: 24.4 -> 27.5 (+3.1) | Helped |
| add VIGC coco data to MiniGPT-4+ training | MiniGPT-4+ · LLaVA-Eval | All relative score: 84.7 -> 87.0 (+2.3) | Helped |
| add VIGC generated data to InstructBLIP fine-tuning | InstructBLIP · OKVQA | accuracy: 63.1 -> 63.8 (+0.7) | Helped |
| add VIGC generated data to InstructBLIP fine-tuning | InstructBLIP · A-OKVQA | accuracy: 62.5 -> 64.1 (+1.6) | Helped |
| add VIGC generated data to MiniGPT-4+ fine-tuning | MiniGPT-4+ · OKVQA | accuracy: 59.1 -> 59.8 (+0.7) | Helped |
Surprising findings
- Adding VIG-generated detailed descriptions without correction gave only marginal gains (81.4 vs 81.0) due to hallucinations, while VIC correction raised it to 85.8.
- VIC correction was reported to be less effective for dialogue and inference tasks than for detailed descriptions.
Limitations
- Some hallucinations persist after VIC correction (10 cases remain in the evaluated sample).
- Evaluation relies on GPT-4 relative scoring on only 90 questions from 30 images.
- Improvements on OKVQA/A-OKVQA are modest (0.6-1.6 points).
Experimental setup
Datasets: LLaVA-150K OKVQA A-OKVQA COCO 2017 Objects365 MMBench
Benchmarks: LLaVA-Eval MMBench OKVQA A-OKVQA
Metrics: relative score vs GPT-4 accuracy hallucination count
Baselines: LLaVA-7B LLaVA-13B MiniGPT-4+ InstructBLIP PaLM-E PromptCap
Builds on: MiniGPT-4 LLaVA InstructBLIP BLIP-2
Closest papers by meaning
- Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain (ICML 2026 · similarity 0.91)
- Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.90)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.90)
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.90)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
- Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.