Research map / Vision-language and multimodal / CLIP and zero-shot
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMICL, a VLM with an image-declaration context scheme and a multi-modal in-context tuning dataset (MIC) to handle interleaved multi-image prompts, achieving SOTA zero-shot MME/MMBench results and better multi-image ICL.
Problem
Most vision-language models handle only single-image prompts and struggle with complex interleaved multi-image prompts, text-to-image references, and in-context multi-modal demonstrations, and suffer language bias.
Approach
Treat image and text tokens equally in an interleaved sequence, add textual image declarations with unique image proxy tokens to link text and images, and build a 5.8M-sample multi-modal in-context dataset (MIC) with interconnected images from videos and VCR crops, then tune on it in a second stage.
Key results
- MMICL with InstructBLIP-FLAN-T5-XXL achieves total average 129.33 on MME, the best among compared VLMs.
- On Winoground, MMICL (FLAN-T5-XXL) reaches 43.00 group score, above GPT4-V's 39.25.
- On ScienceQA-IMG split, MMICL gets 82.10 average with a performance gap of only 0.90 between groups needing vs not needing visual information.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MMICL instead of BLIVA | VLM evaluation · MME | Total Avg.: 119.23 -> 129.33 | Helped |
| Use MMICL instead of KOSMOS-1 | Zero-shot visual reasoning · RAVEN IQ test | Accuracy: 22.00 -> 34.00 | Helped |
| Add 4 in-context examples | MMICL (Instruct-FLAN-T5-XXL) · VizWiz | score: 24.45 -> 50.28 | Helped |
| Add 4 in-context examples | MMICL (FLAN-T5-XXL) · Flickr 30K | score: 85.03 -> 89.27 | Helped |
| Add multi-modal in-context tuning (Stage II) (ablation) | InstructBLIP-FLAN-T5-XL · VSR | score: 61.54 -> 70.54 | Helped |
| Add the context scheme (ablation) | MMICL with InstructBLIP-FLAN-T5-XL · MME Perception | score: 1238.99 -> 1303.59 | Helped |
| Add image declaration (ablation) | MMICL with InstructBLIP-FLAN-T5-XL · Winoground | Group score: 3.00 -> 38.75 | Helped |
| Add the in-context format (ablation) | MMICL with InstructBLIP-FLAN-T5-XL · NLVR2 | score: 62.63 -> 72.45 | Helped |
| Add interrelated images (ablation) | MMICL with InstructBLIP-FLAN-T5-XL · RAVEN | score: 16.00 -> 32.00 | Helped |
Surprising findings
- GPT4-V outperforms MMICL on Winoground text score (69.25 vs 45.00), showing MMICL's advantage is mainly in group score.
- Few-shot performance on Flickr30K decreases with in-context examples for some models, attributed to caption exemplars adding noise.
Limitations
- Due to resource constraints, only about 10% of the 5.8M-sample dataset was used for fine-tuning.
- Only FLAN-T5 XL/XXL backbones were explored; larger models trained on all data are untested.
- Some baselines are reported from leaderboards, so comparisons are not fully controlled.
Experimental setup
Datasets: MS COCO DiffusionDB Flickr30K MiniImageNet VQAv2 ST-VQA TextVQA NLVR2 RefCOCO OK-VQA GQA VCR
Benchmarks: MME MMBench Winoground RAVEN Flickr30K WebSRC VQAv2 Hateful Memes VizWiz ScienceQA-IMG VSR IconQA
Metrics: accuracy VQA accuracy CIDEr group score total score
Baselines: Flamingo KOSMOS-1 BLIP-2 InstructBLIP Shikra Otter Ying-VLM LLaVA MiniGPT-4 GPT4-V
Builds on: BLIP-2 InstructBLIP Flamingo Otter
Closest papers by meaning
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.92)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.91)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
- What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration (NeurIPS 2024 · similarity 0.90)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.90)
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.