Research map / Vision-language and multimodal / Multimodal LLMs
AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AIM, a light framework that aggregates each demonstration's image information into fused virtual tokens at its text positions, letting single-image MLLMs (Qwen-VL, LLaVA-Next) perform efficient multimodal in-context learning with a frozen backbone and a 17M projection layer.
Problem
Multimodal ICL is hampered because most MLLMs are trained only on single-image data and multi-image demonstrations produce thousands of visual tokens, causing memory blow-up and degraded ICL performance.
Approach
Forward each image-text demonstration independently through the first N LLM layers, keep only the hidden states at the text label positions, and project them with a learned linear layer into fused virtual tokens that replace the image-text pair, so the prompt resembles one query image plus text-like demonstrations.
Key results
- On Flickr30k 16-shot, QWen-based AIM reaches 84.1 CIDEr versus 66.1 for QWen-VL, a gain of about 18 CIDEr.
- AIM retains about 4% of original tokens in LLaVA-based demonstrations and 8% in QWen-based ones on average.
- LLaVA-Next baseline collapses to under 1 metric beyond one demonstration on Flickr30k, while LLaVA-based AIM reaches 53.4 VQA-ACC on OKVQA at 16 shots.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| aggregate demonstration images into text-like fused tokens and drop their visual tokens | QWen-VL · Flickr30k, 16-shot | CIDEr: 66.1 -> 82.3 | Helped |
| aggregate demonstration images into text-like fused tokens and drop their visual tokens | QWen-VL · OKVQA, 16-shot | VQA-ACC: 53.7 -> 56.0 | Helped |
| aggregate demonstration images into text-like fused tokens and drop their visual tokens | QWen-VL · Vizwiz, 16-shot | VQA-ACC: 29.8 -> 36.1 | Helped |
| aggregate demonstration images into text-like fused tokens and drop their visual tokens | QWen-VL · Hateful Memes, 16-shot | ROC-AUC: 58.2 -> 57.1 | Hurt |
| aggregate demonstration images into text-like fused tokens and drop their visual tokens | LLaVA-Next · Flickr30k, 16-shot | CIDEr: <1 -> 53.4 | Helped |
| aggregate demonstration images into text-like fused tokens and drop their visual tokens | LLaVA-Next · OKVQA, 16-shot | VQA-ACC: <1 -> 54.6 | Helped |
| aggregate demonstration images into text-like fused tokens and drop their visual tokens | LLaVA-Next · Vizwiz, 16-shot | VQA-ACC: <1 -> 20.2 | Helped |
| aggregate demonstration images into text-like fused tokens and drop their visual tokens | LLaVA-Next · Hateful Memes, 16-shot | ROC-AUC: NaN -> 55.6 | Helped |
| use the first 16 rather than all LLM layers for image-information aggregation (ablation) | QWen-VL AIM · Flickr30k, 16-shot | CIDEr: 82.3 -> 84.1 | Helped |
| use the first 16 rather than all LLM layers for image-information aggregation (ablation) | LLaVA-Next AIM · Flickr30k, 16-shot | CIDEr: 53.4 -> 26.5 | Hurt |
| use the first 24 rather than all LLM layers for image-information aggregation (ablation) | LLaVA-Next AIM · Flickr30k, 16-shot | CIDEr: 53.4 -> 38.0 | Hurt |
| provide text-only demonstrations instead of visual demonstrations (ablation) | QWen-VL · Flickr30k, 16-shot | CIDEr: 66.1 -> 71.7 | Helped |
Surprising findings
- QWen-VL given text-only demonstrations (w/o visual) outperforms its own many-shot setting with both visual and textual demonstrations.
- LLaVA-based AIM benefits from aggregating over all LLM layers, contrary to the label-words claim that shallow layers aggregate information.
Limitations
- Efficiency gains shrink when label texts are very long, since textual tokens remain.
- Caching demonstration virtual tokens consumes substantial storage as demonstrations grow.
- Lower performance on VizWiz versus other multimodal ICL methods because the model is not prompted to answer 'unanswerable'.
Experimental setup
Datasets: MMC4 Flickr30k OKVQA VizWiz Hateful Memes
Benchmarks: Flickr30k OKVQA VizWiz Hateful Memes
Metrics: CIDEr VQA accuracy ROC AUC accuracy perplexity throughput memory cost
Baselines: Flamingo OpenFlamingo QWen-VL LLaVA-Next MMICL Otter
Builds on: Flamingo OpenFlamingo Qwen-VL LLaVA-Next ICAE
Closest papers by meaning
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.93)
- SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization (CVPR 2025 · similarity 0.91)
- Link-Context Learning for Multimodal LLMs (CVPR 2024 · similarity 0.90)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.90)
- Towards More Unified In-context Visual Understanding (CVPR 2024 · similarity 0.90)
- Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.