Research map / Vision-language and multimodal / Multimodal LLMs

MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MoDA, a lightweight instruction-conditioned channel-wise multiplicative modulation adapter inserted after the MLLM vision adapter, evaluated on LLaVA-1.5, LLaVA-MoRE and Qwen3-VL across 12 benchmarks.

Problem

MLLMs struggle with fine-grained visual grounding because patch embeddings from visual encoders entangle multiple semantic elements, leading to hallucinations and errors on detail-focused queries.

Approach

MoDA uses cross-attention between language token embeddings and pre-aligned visual features to produce a sigmoid channel-wise mask applied multiplicatively (Hadamard product) to visual embeddings, giving instruction-conditioned feature-dimension selection without architectural changes or extra supervision.

Key results

• On MMVP, LLaVA-1.5 (Vicuna-7B) with MoDA improves from 24.0 to 36.0 (+12.0 points).
• LLaVA-MoRE SigLIP-S2 + MoDA improves ScienceQA from 77.1 to 81.9 and MMVP from 39.3 to 42.7 with ~3.7% extra parameters and <1% FLOPs.
• On Qwen3-VL-2B, MoDA improves GQA by 3.8, ScienceQA by 4.9, RealWorldQA by 4.1, but regresses on ChartQA (80.0 to 79.0) and V*Bench (77.0 to 74.9).

Surprising findings

• Adding an L1 auxiliary loss consistently degraded cross-attention MoDA performance across both encoders.
• Applying MoDA at all LLM layers gave smaller gains (+0.7 avg) than injecting it only at the start of the LLM (+2.6 avg).

Limitations

• Soft channel re-weighting cannot produce explicit channel sparsity (channels rarely set to zero).
• Gains depend on encoder underutilizing task-relevant channels; regressions observed on ChartQA and V*Bench for Qwen3-VL.
• Evaluation limited to single-image benchmarks; multi-image and video not tested.

Experimental setup

Datasets: LLaVA 558K caption pretraining data LLaVA visual instruction tuning data

Benchmarks: GQA ScienceQA MMBench RealWorldQA ChartQA LLaVA-Wild MM-Vet MMStar V*Bench CV-Bench POPE MMVP

Metrics: accuracy average score

Baselines: LLaVA-1.5 LLaVA-MoRE Qwen3-VL-2B-Instruct InstructBLIP BLIP-2 ShareGPT-4V Learnable Mask Sparse Mask

Builds on: LLaVA-1.5 Q-Former InstructBLIP LAM LLaVA-MoRE

Closest papers by meaning

• Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.92)
• Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.91)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.90)
• Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.90)
• Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.90)
• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.90)