Research map / Vision-language and multimodal / Multimodal LLMs
Visual Grounding First, Multimodal In-context Learning Follows
Minhyuk Seo, Minjae Lee, Chaeeun Lee, Wei Lin, Muhammad Jehanzeb Mirza, Tinne Tuytelaars, Jonghyun Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper studies multimodal in-context learning (ICL) in MLLMs and argues that visual grounding should come before ICL, rather than training on ICL-specific datasets or extracting task vectors.
Problem
Extending in-context learning to multimodal LLMs is hard; prior multimodal ICL methods rely on ICL-specific datasets or task vector extraction, which improve only the adapted benchmark, generalize poorly, and cause forgetting of previously learned abilities.
Approach
Improve multimodal ICL by first grounding the model's visual understanding, with ICL ability then following, instead of task-specific ICL training or task vector extraction. Details are not available in the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SFT on contrastive pairs with proposition-level contrastive RL (ablation) | Qwen-VL-3-8B · MS COCO ICL captioning 4-shot | ROUGE-L: 26.84 -> 46.26 | Improved |
| Use proposition-level instead of sentence-level contrastive reward (ablation) | Qwen-VL-3-8B · MS COCO ICL captioning 4-shot | ROUGE-L: 44.11 -> 46.26 | Improved |
| Use proposition-level instead of sentence-level contrastive reward (ablation) | Qwen-VL-3-8B · PICL captioning 4-shot | ROUGE-L: 33.41 -> 36.76 | Improved |
| Train with SFT on contrastive pairs (vs. off-the-shelf) (ablation) | Qwen-VL-3-8B · PICL captioning 4-shot | ROUGE-L: 29.29 -> 26.81 | Hurt |
| Train ICL data-dependent methods on one ICL benchmark, test on another | MT-ICL / STV · TrueMICL 4-shot (trained on ISEKAI) | accuracy: 55.03 -> 49.89 | Hurt |
| Add retrieval and ordering of demonstrations (MMICES) | Qwen-VL-3-8B · PICL captioning 2-shot | ROUGE-L: 24.78 -> 21.70 | Hurt |
| Add hallucination-mitigation (e.g. PAI) instead of visual-grounding reward | Qwen-VL-3-8B · MS COCO ICL captioning 0-shot | ROUGE-L: 15.53 -> 13.92 | Hurt |
| Add proposition-level contrastive reward training | Qwen-VL-3-8B · MS COCO ICL captioning 4-shot | ROUGE-L: 23.52 -> 46.26 | Improved |
| Add proposition-level contrastive reward training | Qwen-VL-3-8B · TrueMICL 8-shot | accuracy: 62.70 -> 68.08 | Improved |
| Add proposition-level contrastive reward training | Qwen-VL-3-8B · PICL personalized identifier accuracy 1-shot | identifier accuracy (%): 38 -> 62 | Improved |
| Weight text reward above image reward (lambda_T > lambda_I) (ablation) | Proposition-level contrastive reward · MS COCO captioning | accuracy heatmap | Hurt |
| Use small open-source LLM (Qwen3-1.7B, LLaMA3.2-3B) for proposition decomposition instead of GPT-4o-mini (ablation) | Proposition-level contrastive reward · ICL captioning | performance | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SPARC — SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models (CVPR 2025) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.91)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.91)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.89)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.89)
- Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning (AAAI 2026 · similarity 0.88)
- Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
