AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Visual Grounding First, Multimodal In-context Learning Follows

Minhyuk Seo, Minjae Lee, Chaeeun Lee, Wei Lin, Muhammad Jehanzeb Mirza, Tinne Tuytelaars, Jonghyun Choi

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

The paper studies multimodal in-context learning (ICL) in MLLMs and argues that visual grounding should come before ICL, rather than training on ICL-specific datasets or extracting task vectors.

Problem

Extending in-context learning to multimodal LLMs is hard; prior multimodal ICL methods rely on ICL-specific datasets or task vector extraction, which improve only the adapted benchmark, generalize poorly, and cause forgetting of previously learned abilities.

Approach

Improve multimodal ICL by first grounding the model's visual understanding, with ICL ability then following, instead of task-specific ICL training or task vector extraction. Details are not available in the abstract.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace SFT on contrastive pairs with proposition-level contrastive RL (ablation)Qwen-VL-3-8B · MS COCO ICL captioning 4-shotROUGE-L: 26.84 -> 46.26Improved
Use proposition-level instead of sentence-level contrastive reward (ablation)Qwen-VL-3-8B · MS COCO ICL captioning 4-shotROUGE-L: 44.11 -> 46.26Improved
Use proposition-level instead of sentence-level contrastive reward (ablation)Qwen-VL-3-8B · PICL captioning 4-shotROUGE-L: 33.41 -> 36.76Improved
Train with SFT on contrastive pairs (vs. off-the-shelf) (ablation)Qwen-VL-3-8B · PICL captioning 4-shotROUGE-L: 29.29 -> 26.81Hurt
Train ICL data-dependent methods on one ICL benchmark, test on anotherMT-ICL / STV · TrueMICL 4-shot (trained on ISEKAI)accuracy: 55.03 -> 49.89Hurt
Add retrieval and ordering of demonstrations (MMICES)Qwen-VL-3-8B · PICL captioning 2-shotROUGE-L: 24.78 -> 21.70Hurt
Add hallucination-mitigation (e.g. PAI) instead of visual-grounding rewardQwen-VL-3-8B · MS COCO ICL captioning 0-shotROUGE-L: 15.53 -> 13.92Hurt
Add proposition-level contrastive reward trainingQwen-VL-3-8B · MS COCO ICL captioning 4-shotROUGE-L: 23.52 -> 46.26Improved
Add proposition-level contrastive reward trainingQwen-VL-3-8B · TrueMICL 8-shotaccuracy: 62.70 -> 68.08Improved
Add proposition-level contrastive reward trainingQwen-VL-3-8B · PICL personalized identifier accuracy 1-shotidentifier accuracy (%): 38 -> 62Improved
Weight text reward above image reward (lambda_T > lambda_I) (ablation)Proposition-level contrastive reward · MS COCO captioningaccuracy heatmapHurt
Use small open-source LLM (Qwen3-1.7B, LLaMA3.2-3B) for proposition decomposition instead of GPT-4o-mini (ablation)Proposition-level contrastive reward · ICL captioningperformanceNo clear effect

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy