Research map / Vision-language and multimodal / Multimodal LLMs

Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CAMA, a training-free, plug-and-play method that modulates LVLM self-attention logits in shallow layers (intra-ICD grounding) and middle layers (query-centric routing) to improve multimodal in-context learning.

Problem

Multimodal in-context learning in LVLMs is unstable even with well-matched demonstrations, because the models under-use the in-context sequence; existing fixes rely on prompt engineering, data-heavy tuning, or contrastive decoding with extra forward passes.

Approach

Analyze attention in interleaved multi-image ICL and identify two deficits (weak image-text grounding within each ICD in shallow layers; poor query-to-ICD routing in middle layers), then fix them at inference time by adding context-aware logit biases to selected image tokens (via anchor-token attention differences) and to query-centric heads (via query-ICD embedding similarity), with a position-de

Key results

• CAMA achieves the highest accuracy in all 28 model-benchmark settings (8-shot), averaging +2.96% over vanilla across four LVLMs.
• CAMA combined with VE (CAMA(VE)) reaches 73.63 average on Qwen2.5VL-7B vs 69.85 vanilla and 71.32 SoFA.
• CAMA improves 8-shot average accuracy over vanilla by 2.15% to 6.52% across 2-16 shots.

Surprising findings

• Prompt-based instruction (+Inst) alone gives only marginal gains, but combined with CAMA it yields larger improvements, suggesting attention deficits block prompt benefits.

Limitations

• Adds inference latency relative to vanilla decoding (about 5% in reported setting).
• Performance degrades when the context contains many irrelevant or misleading ICDs.
• Requires manual bounding-box annotation for the attention-deficit analysis; hyperparameters (layers, k) selected on the evaluated benchmarks.

Experimental setup

Datasets: VQAv2 VizWiz OK-VQA GQA TextVQA CLEVR (VL-ICL bench) MMStar Flickr30k MSCOCO Hatefulmemes LLaVA-Interleave Bench visual storytelling

Benchmarks: VQAv2 VizWiz OK-VQA GQA TextVQA CLEVR MMStar Flickr30k MSCOCO Hatefulmemes L-I-VST

Metrics: accuracy CIDEr ROC-AUC L-I-score

Baselines: +Inst Contrastive decoding (CD) Visual enhancement (VE) SoFA SigLIP-based retrieval (TACO)

Builds on: SoFA contrastive decoding visual enhancement

Closest papers by meaning

• VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.90)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.89)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.89)
• A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.89)
• AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.89)