Research map / Vision-language and multimodal / CLIP and zero-shot
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| instruction-tune the BLIP-2 backbone on diverse vision-language instruction data | BLIP-2 (FlanT5-XL) · NoCaps | CIDEr: 107.5 -> 119.9 | Helped |
| make visual feature extraction instruction-aware in the Q-Former (ablation) | InstructBLIP (FlanT5-XL) · ScienceQA (image-context) | top-1 accuracy (%): 63.4 -> 70.4 | Helped |
| make visual feature extraction instruction-aware in the Q-Former (ablation) | InstructBLIP (Vicuna-7B) · iVQA | accuracy: 36.8 -> 52.2 | Helped |
| use balanced data sampling (ablation) | InstructBLIP (FlanT5-XL) · Held-in average | average score: 92.6 -> 94.1 | Helped |
| use balanced data sampling (ablation) | InstructBLIP (Vicuna-7B) · Held-in average | average score: 98.8 -> 100.8 | Helped |
| train with natural-language instructions rather than multitask input-output formats | BLIP-2 (FlanT5-XL) · held-out datasets | average score | Helped |
Closest papers by meaning
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.90)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.90)
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.90)
- Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning (CVPR 2024 · similarity 0.89)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.89)
- Language-Instructed Vision Embeddings for Controllable and Generalizable Perception (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.