Research map / Vision-language and multimodal / Human and embodied modeling
What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic empirical study of factors affecting multi-modal in-context learning (MM-ICL) across demonstration retrieval, ordering, and prompt construction, using 6 vision LLMs and 20 strategies on 4 tasks.
Problem
The underlying factors that determine MM-ICL effectiveness remain under-explored, as prior work mostly focuses on optimizing MM-ICL rather than understanding what drives its performance.
Approach
Decomposes the MM-ICL prompt pipeline into demonstration retrieval (sample representation, comparison, selection), demonstration ordering (intra- and inter-demonstration), and prompt construction (delimiters, instructions, shot count), and measures each factor's effect across several VLLMs and tasks from M3IT and M3CoT.
Key results
- Multi-modal retriever achieves the highest average score for most models (e.g., Qwen-VL 62.22 avg vs 58.87 random few-shot; Gemini-Pro 68.16 vs 63.75).
- Model size from 8B to over 100B does not significantly improve MM-ICL performance, suggesting multi-modal alignment matters more than scale.
- Introductory instructions consistently improve performance, while summative and intra-demonstration instructions generally decrease it; more than three demonstrations often hurts performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-modal rather than single-modal retrieval | MM-ICL retrieval · Four tasks across six VLLMs | AVG | Helped |
| Use textual retrieval rather than multi-modal retrieval | MM-ICL retrieval · Classification and reasoning | task performance | Mixed |
| Use cosine similarity rather than L2 similarity | MM-ICL sample comparison · MM-ICL tasks | task performance | Helped |
| Select demonstrations for semantic diversity | MM-ICL retrieval · MM-ICL tasks | task performance | No clear effect |
| Include out-of-domain demonstrations | MM-ICL retrieval · MM-ICL | average performance: nearly 4% performance drop | Hurt |
| Favor stylistically similar images in retrieved demonstrations | MM-ICL retrieval · VQA, captioning, classification, and reasoning | task performance | Mixed |
| Put the image first within each demonstration | MM-ICL demonstration ordering · MM-ICL | model performance | Helped |
| Change inter-demonstration order (random, similar-last, or similar-first) | MM-ICL demonstration ordering · MM-ICL | model performance | No clear effect |
| Add an introductory instruction before demonstrations | MM-ICL prompt · MM-ICL tasks | model performance | Helped |
| Use summative or intra-demonstration instructions instead of introductory instructions | MM-ICL prompt · MM-ICL tasks | model performance | Hurt |
| Increase the number of demonstrations | MM-ICL prompt · Caption and VQA | task performance | Mixed |
| Increase the number of demonstrations | MM-ICL prompt · Reasoning | task performance | No clear effect |
Surprising findings
- Increasing model size from 8B to over 100B does not significantly enhance MM-ICL performance.
- Inter-demonstration ordering has a negligible effect, while diversity-based selection gives limited gains in MM-ICL unlike text-only ICL.
Limitations
- Restricted to image and text modalities; video and multilingual MM-ICL not explored.
- Certain image instructions such as grounding or arrows are not considered.
- Experiments use 500 sampled items per task due to cost constraints.
Experimental setup
Datasets: M3IT M3CoT OKVQA VQA-v2
Benchmarks: M3IT M3CoT
Metrics: CIDEr BERTScore Token F1 Accuracy F1 RAS
Baselines: Random few-shot Textual Retriever Visual Retriever Zero-shot
Builds on: M3IT M3CoT BridgeTower CLIP
Closest papers by meaning
- MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (ICLR 2024 · similarity 0.90)
- Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning (CVPR 2025 · similarity 0.90)
- Mixture of Demonstrations for In-Context Learning (NeurIPS 2024 · similarity 0.90)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.89)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.89)
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.