Research map / Vision-language and multimodal / Visual question answering
VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VL-ICL Bench, a 10-task benchmark for multimodal in-context learning spanning image-to-text and text-to-image tasks, and evaluates ~20 vision-language models showing that current VLLMs struggle to exploit many shots.
Problem
Existing multimodal ICL evaluations rely on few-shot VQA and captioning, which mainly teach answer formatting rather than true in-context learning, so the broader capabilities and limits of VLLM in-context learning are under-explored.
Approach
Replace VQA/captioning-based ICL evaluation with a suite of tasks that require binding new symbols, rule/operator induction, counting, OCR, cross-image relation learning, and image generation from few-shot examples, and measure zero-shot accuracy, peak accuracy and ICL efficiency as functions of shots.
Key results
- LLaVA-OneVision-72B achieves the best average image-to-text ICL, with peak accuracy of 98.7% on Fast Open-Ended MiniImageNet and 75.6% on Operator Induction, closely followed by GPT-4V (92.0% peak on Operator Induction).
- On VQA and captioning benchmarks, switching from exact match to LLM-based evaluation almost eliminates the benefit of ICL over zero-shot, indicating those benchmarks mainly teach answer format.
- Extending context with SelfExtend (4k to 16k) helps some tasks but does not consistently improve ICL, suggesting long-context capability is necessary but not sufficient.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use image inputs instead of semantically equivalent text inputs | image-to-text VL-ICL tasks · CLEVR Count Induction, Operator Induction, and Interleaved Operator Induction | accuracy | Hurt |
| Extend context length from 4k to 16k with SelfExtend (ablation) | LLaVA-Next-7B · Fast Open-Ended MiniImageNet | peak accuracy: 37.2 -> 51.0 | Helped |
| Extend context length from 4k to 16k with SelfExtend (ablation) | LLaVA-Next-7B · CLEVR Count Induction | peak accuracy: 25.2 -> 29.0 | Helped |
| Extend context length from 4k to 16k with SelfExtend (ablation) | LLaVA-Next-7B · Operator Induction | peak accuracy: 10.6 -> 11.7 | Helped |
| Extend context length from 4k to 16k with SelfExtend (ablation) | LLaVA-Next-7B · TextOCR | peak accuracy: 24.7 -> 26.0 | Helped |
| Extend context length with SelfExtend (ablation) | VILA-7B · Operator Induction | peak accuracy: 28.3 -> 28.3 | No clear effect |
| Use similarity-based demonstration selection instead of random selection (ablation) | image-to-text and text-to-image models · CLEVR Count Induction, TextOCR, and CoBSAT | peak accuracy: ~3% improvement | Helped |
| Use chain-of-thought prompting (ablation) | Qwen-VL-Chat and InternLM-XComposer2 · VL-ICL Bench tasks | accuracy | Mixed |
| Duplicate support examples (ablation) | Qwen-VL-Chat · Fast Open-Ended MiniImageNet | accuracy | Helped |
| Duplicate support examples (ablation) | Qwen-VL-Chat · VL-ICL Bench tasks other than Fast Open-Ended MiniImageNet | accuracy | Mixed |
| Use detailed rather than concise or absent task descriptions (ablation) | VLLMs · VL-ICL Bench tasks | accuracy | Mixed |
| Increase model size from 0.5B or 7B to 72B (ablation) | LLaVA-OneVision · Interleaved Operator Induction and Fast Matching MiniImageNet | accuracy | Helped |
Surprising findings
- Zero-shot performance is not strongly indicative of ICL ability; LLaVA-Next-7B is among the weakest on the benchmark despite strong mainstream zero-shot results.
- Many models show negative ICL efficiency, with performance decreasing as shots increase beyond a few examples.
Limitations
- Only a small number of text-to-image models are evaluated compared with image-to-text models.
- Some evaluation relies on an LLM judge (LLaVA-Next-7B) for text-to-image correctness, which may be imperfect.
- Models with limited context windows and image counts cannot be fairly tested at high shot counts.
Experimental setup
Datasets: MiniImageNet CLEVR TextOCR CoBSAT MathVista VizWiz COCO Captioning
Benchmarks: VL-ICL Bench MathVista VizWiz COCO Captioning
Metrics: accuracy zero-shot accuracy peak accuracy ICL efficiency
Baselines: GPT-4V LLaVA-Next-7B IDEFICS-80B Mantis-Idefics2 InternLM-XComposer2.5
Builds on: MiniImageNet CLEVR CoBSAT SelfExtend
Closest papers by meaning
- LIVE: Learnable In-Context Vector for Visual Question Answering (NeurIPS 2024 · similarity 0.93)
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.93)
- MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (ICLR 2024 · similarity 0.91)
- How to Configure Good In-Context Sequence for Visual Question Answering (CVPR 2024 · similarity 0.91)
- Towards More Unified In-context Visual Understanding (CVPR 2024 · similarity 0.91)
- Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.