Research map / Vision-language and multimodal / Human and embodied modeling
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compresses many multimodal in-context examples into attention-head activation vectors (Multimodal Task Vectors, MTV) extracted via a REINFORCE-style head search, enabling many-shot ICL in LMMs without finetuning or extra context tokens.
Problem
Many-shot multimodal in-context learning is limited by LMM context length, since interleaved image tokens are expensive, and long-context inference is slow and memory-heavy.
Approach
Decouple task-vector extraction into (1) averaging last-token attention-head activations over many few-shot inference calls to encode many shots, and (2) separately searching for which heads to patch using a Bernoulli-policy REINFORCE-style procedure on format-matched downstream examples; the patched activations then replace head outputs at inference with no extra context tokens.
Key results
- On VizWiz, MTV raises Qwen-VL-7B from 35.2 to 45.6 (vs 44.3 for 8-shot ICL) and Idefics2 from 31.3 to 52.5, with VILA-1.5-8B reaching 55.2 from 28.0.
- On unseen-class object classification, MTV+1-shot ICL improves Qwen-VL-7B on Flowers from 55.0 to 78.1 and on CUB from 56.5 to 80.0.
- MTV on Qwen-VL with 4-shot ICL stacked on VizWiz reaches 58.1 with high-quality facility-location shots, compared with 45.6 for plain MTV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MTV instead of zero-shot inference | Qwen-VL-7B · VizWiz validation | accuracy: 35.2 -> 45.6 | Helped |
| apply MTV instead of few-shot ICL | Idefics2 · VizWiz validation | accuracy: 43.8 -> 52.5 | Helped |
| apply MTV instead of few-shot ICL | VILA-1.5-8B · OK-VQA validation | accuracy: 36.5 -> 40.6 | Helped |
| apply MTV instead of few-shot ICL | MANTIS-LLaMA3-8B · VizWiz validation | accuracy: 27.5 -> 51.0 | Helped |
| add MTV alongside 1-shot ICL | Qwen-VL-7B · Flowers | accuracy: 55.0 -> 78.1 | Helped |
| add MTV alongside 1-shot ICL | VILA-1.5-8B · CUB | accuracy: 88.4 -> 89.7 | Helped |
| fine-tune shots by Facility Location instead of selecting random shots (ablation) | QwenVL-7B + MTV · VizWiz validation | accuracy: 45.6 -> 58.1 | Helped |
| increase iterations from 100 to 200 at 25 shots per iteration (ablation) | QwenVL-7B + MTV · VizWiz validation | accuracy: 56.4 -> 51.4 | Hurt |
| use 4 shots for 1 iteration to calculate mean activations (ablation) | ViLA-1.5-8B + MTV · VizWiz validation | accuracy: 39.3 -> 57.4 | Helped |
| reuse attention heads extracted on VizWiz for OK-VQA | ViLA-1.5-8B + MTV · OK-VQA validation | accuracy: 36.5 -> 38.3 | Helped |
| encode 400 examples with MTV instead of using 16-shot ICL | ViLA-1.5-8B · VizWiz validation | Runtime per 100 iterations (min): 3.3 -> 1.9 | Helped |
| encode 400 examples with MTV instead of using 16-shot ICL | ViLA-1.5-8B · VizWiz validation | Max GPU Memory (GB): 20.6 -> 19.8 | Helped |
Surprising findings
- Vanilla ICL on MANTIS-LLaMA3-8B hurts VizWiz (36.3 zero-shot vs 26.4 at 4-shot), while MTV reaches 51.0.
- Finetuning on VizWiz improves VizWiz (62.0) but drops OK-VQA sharply (58.6 to 25.1), indicating forgetting.
Limitations
- Requires access to model internals, so it cannot be applied to proprietary models such as GPT-4 or Gemini.
- Gains are shown only on a small set of VL tasks and a few open models; the paper itself describes document and language results as preliminary.
- Performance saturates with more examples and depends on task, with the paper noting no general scaling law.
Experimental setup
Datasets: VizWiz OK-VQA Flowers CUB-200-2011 ChartQA TextVQA
Benchmarks: VizWiz OK-VQA Flowers CUB-200-2011 ChartQA TextVQA
Metrics: accuracy
Baselines: few-shot ICL (4-shot, 8-shot) Function Vectors Visual Task Vectors VizWiz finetuning Flamingo 9B BLIP3
Builds on: function vectors visual task vectors REINFORCE
Closest papers by meaning
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.89)
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.89)
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.89)
- Link-Context Learning for Multimodal LLMs (CVPR 2024 · similarity 0.89)
- Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning (AAAI 2026 · similarity 0.89)
- Many-Shot In-Context Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.