Research map / Vision-language and multimodal / Multimodal LLMs
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Aligns a frozen BLIP-2 ViT/Q-Former with a frozen Vicuna LLM via a single linear projection, trained in two stages (caption pretraining, then curated detailed-description finetuning) to elicit GPT-4-like vision-language abilities.
Problem
Vision-language models lack advanced multimodal abilities seen in GPT-4 (detailed description, website creation from sketches, humor explanation); the paper hypothesizes these stem from aligning visual features with a more advanced LLM.
Approach
Align frozen visual encoder (EVA-CLIP ViT-G/14 + Q-Former from BLIP-2) with frozen Vicuna using only one linear projection layer, then a second-stage finetune on ~3,500 self-curated, ChatGPT-cleaned detailed image descriptions to fix repetition and fragmentation from caption-only pretraining.
Key results
- MiniGPT-4 satisfies 65/100 advanced-task requests versus 5/100 for BLIP-2 (human evaluation).
- On COCO caption coverage judged by ChatGPT, MiniGPT-4 reaches 66.2% versus 27.5% for BLIP-2.
- Second-stage finetuning reduces detailed-caption failure rate from 35% to 2% and poem failure from 32% to 1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MiniGPT-4 with Vicuna instead of BLIP-2 with Flan-T5 XXL | vision-language model · COCO caption evaluation | Percentage: 27.5% -> 66.2% | Helped |
| use MiniGPT-4 instead of BLIP-2 | vision-language model · Advanced vision-language tasks: Avg. | requests satisfied: 5/100 -> 65/100 | Helped |
| add second-stage finetuning with curated detailed image descriptions (ablation) | MiniGPT-4 · Detailed caption generation | Failure rate: 35% -> 2% | Helped |
| add second-stage finetuning with curated detailed image descriptions (ablation) | MiniGPT-4 · Poem generation | Failure rate: 32% -> 1% | Helped |
| remove the Q-Former (ablation) | MiniGPT-4 · AOK-VQA | score: 58.2 -> 56.9 | No clear effect |
| remove the Q-Former (ablation) | MiniGPT-4 · GQA | score: 32.2 -> 33.4 | No clear effect |
| replace one projection layer with three linear layers (ablation) | MiniGPT-4 · AOK-VQA | score: 58.2 -> 49.7 | Hurt |
| replace one projection layer with three linear layers (ablation) | MiniGPT-4 · GQA | score: 32.2 -> 31.0 | Hurt |
| add Q-Former finetuning (ablation) | MiniGPT-4 · AOK-VQA | score: 58.2 -> 52.1 | Hurt |
| add Q-Former finetuning (ablation) | MiniGPT-4 · GQA | score: 32.2 -> 28.0 | Hurt |
| unfreeze Vicuna with LoRA and add VQAv2, OKVQA, and A-OKVQA data during stage-2 (ablation) | MiniGPT-4 · AOK-VQA | score: 58.2 -> 67.2 | Helped |
| unfreeze Vicuna with LoRA and add VQAv2, OKVQA, and A-OKVQA data during stage-2 (ablation) | MiniGPT-4 · GQA | score: 32.2 -> 43.5 | Helped |
Surprising findings
- BLIP-2 finetuned on the same second-stage data still fails to generalize to advanced tasks, suggesting the stronger LLM matters more than the data.
- Removing the Q-Former gives similar performance (AOK-VQA 56.9 vs 58.2), suggesting the Q-Former is not critical for advanced skills.
Limitations
- Hallucination increases with longer descriptions (CHAIR_i 9.6 for long vs 7.2 for short outputs).
- Weak spatial information understanding, e.g., failing to locate windows.
- Weaker traditional VQA benchmark performance than BLIP-2 (AOK-VQA 58.2 vs 80.2).
Experimental setup
Datasets: Conceptual Captions SBU LAION COCO A-OKVQA GQA VQAv2 OKVQA Localized Narratives
Benchmarks: COCO caption advanced vision-language tasks (meme, recipe, ad, poem) A-OKVQA GQA CHAIR
Metrics: human success rate ChatGPT-judged caption coverage failure rate accuracy CHAIR_i average length
Baselines: BLIP-2 BLIP-2 FT MiniGPT-4 w/o Q-Former MiniGPT-4 LocNa
Builds on: BLIP-2 Vicuna LLaMA
Closest papers by meaning
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.89)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.88)
- Modality-Specialized Synergizers for Interleaved Vision-Language Generalists (ICLR 2025 · similarity 0.88)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.88)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.88)
- GMAIL: Generative Modality Alignment for generated Image Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.