Research map / Vision-language and multimodal / Multimodal LLMs

Reallocating Attention Across Layers to Reduce Multimodal Hallucination

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a training-free plugin that identifies perception-oriented (shallow) and reasoning-oriented (deep) attention heads in multimodal reasoning models via visual attention ratios and rescales their outputs with fixed gains to reduce hallucination.

Problem

Hallucinations in multimodal large reasoning models arise from imbalanced allocation of attention between perception (shallow layers) and reasoning (deep layers), causing perceptual bias and reasoning drift.

Approach

Classify attention heads by visual attention ratio restricted to shallow (perception) and deep (reasoning) layer ranges, then amplify their outputs with multiplicative gains before the output projection, without retraining or changing the attention mechanism.

Key results

• Average 4.2-point accuracy gain over vanilla across three MLRMs and five benchmarks, per the paper.
• On HallusionBench, inference time rises from ~101s to ~103s per batch (+2.0%), versus +21.7% to +68.3% for AGLA and VCD and +557% for CGD.
• Gains on Ocean-R1 include MathVista 54.58 to 59.32 and SEED-Bench 59.76 to 66.51 accuracy.

Surprising findings

• Enhancing only reasoning heads or only perception heads gives task-skewed and non-additive effects, with some single-group ablations hurting performance while combined rescaling helps.
• Performance peaks occur in bands of layer boundaries that differ by task, rather than at a single crisp split.

Limitations

• Head classification relies on hand-set thresholds and layer boundaries that vary by task and model, selected on evaluation results.
• Theoretical justification for selective enhancement rests on an assumed non-harmful heads property that is not empirically verified.
• Evaluated only on three MLRMs with 7B-16B parameters and five benchmarks.

Experimental setup

Datasets: MathVista MathVision HallusionBench MMStar SEED-Bench

Benchmarks: MathVista MathVision HallusionBench MMStar SEED-Bench

Metrics: accuracy F1 inference time

Baselines: VCD CGD AGLA

Builds on: contrastive decoding (VCD) AGLA CGD

Closest papers by meaning

• More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models (NeurIPS 2025 · similarity 0.93)
• See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs (CVPR 2026 · similarity 0.91)
• Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models (CVPR 2026 · similarity 0.90)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.90)
• Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models (CVPR 2026 · similarity 0.90)
• MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM (NeurIPS 2025 · similarity 0.89)