Research map / Architectures and efficiency / Attention and transformers
Mimic In-Context Learning for Multimodal Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MimIC, lightweight per-head learnable shift vectors with query-dependent magnitude and layer-wise alignment loss that mimic in-context learning in LMMs without demonstrations at inference.
Problem
In-context learning in large multimodal models is highly sensitive to demonstration configurations and costly in context length; existing shift-vector methods use incomplete approximations of the ICD effect.
Approach
Derives that ICDs act as a shift on query hidden states after attention, and approximates this with per-attention-head learnable shift vectors scaled by a query-dependent magnitude from a linear layer, trained with a layer-wise alignment loss to match ICL hidden states.
Key results
- On Idefics-9b, MimIC improves over 32-shot ICL by about 3.5 accuracy points on VQA and 9.00 CIDEr on captioning on average.
- MimIC reaches 59.64 VQAv2 and 52.05 OK-VQA on Idefics-9b, versus 56.18 and 48.48 for 32-shot ICL, with 0.26M added parameters versus 25M for LoRA.
- On Idefics-9b, MimIC surpasses 32-shot ICL with only 200 training samples, while LIVE needs more data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace head-specific shift magnitudes with one magnitude shared across heads (ablation) | MimIC on Idefics1 · VQAv2 | accuracy: 57.89 -> 59.64 | Hurt |
| replace head-specific shift magnitudes with one magnitude shared across heads (ablation) | MimIC on Idefics1 · OK-VQA | accuracy: 50.86 -> 52.05 | Hurt |
| replace head-specific shift magnitudes with one magnitude shared across heads (ablation) | MimIC on Idefics1 · COCO Caption | CIDEr: 111.98 -> 114.89 | Hurt |
| use a fixed shift magnitude instead of query-dependent magnitudes (ablation) | MimIC on Idefics1 · VQAv2 | accuracy: 57.95 -> 59.64 | Hurt |
| use a fixed shift magnitude instead of query-dependent magnitudes (ablation) | MimIC on Idefics1 · OK-VQA | accuracy: 50.94 -> 52.05 | Hurt |
| use a fixed shift magnitude instead of query-dependent magnitudes (ablation) | MimIC on Idefics1 · COCO Caption | CIDEr: 112.48 -> 114.89 | Hurt |
| replace layer-wise L2 alignment loss with KL divergence (ablation) | MimIC on Idefics1 · VQAv2 | L2 distance to 32-shot ICL: 32.13 -> 30.17 | Hurt |
| replace layer-wise L2 alignment loss with KL divergence (ablation) | MimIC on Idefics1 · OK-VQA | L2 distance to 32-shot ICL: 29.76 -> 28.25 | Hurt |
| train MimIC with fewer demonstrations (ablation) | MimIC on Idefics1 · COCO Caption | CHAIRs: 5.57 -> 4.09 | Helped |
Surprising findings
- Retrieval-based RICES underperforms random ICD selection on Idefics2, OK-VQA and COCO, showing ICD selection strategies are model-dependent.
- Non-trainable shift methods (Task Vector, Function Vector) fail to beat zero-shot on Idefics2.
Limitations
- Hallucinations increase with shot count and are slightly higher than zero-shot on captioning.
- Experiments limited to two LMMs and three tasks.
- Idefics2 uses only up to 8-shot ICL due to compute constraints.
Experimental setup
Datasets: VQAv2 OK-VQA COCO Caption
Benchmarks: VQAv2 OK-VQA COCO Caption
Metrics: accuracy CIDEr CHAIR recall L2 distance
Baselines: 32-shot ICL RICES Task Vector Function Vector LIVE LoRA Zero-shot
Builds on: Task Vector Function Vector LIVE LoRA
Closest papers by meaning
- What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration (NeurIPS 2024 · similarity 0.87)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.87)
- LIVE: Learnable In-Context Vector for Visual Question Answering (NeurIPS 2024 · similarity 0.87)
- Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning (AAAI 2026 · similarity 0.86)
- Is In-Context Learning Sufficient for Instruction Following in LLMs? (ICLR 2025 · similarity 0.86)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.