Research map / Vision-language and multimodal / Multimodal LLMs

Beyond Attention Imbalance: Mitigating Hallucinations via Spectral Surgery

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FLASH, a training-free, contrastive-decoding-free method that selects vision heads via a spectral vortex score and applies DCT-domain dual-stream spectral modulation to value matrices and attention scores to reduce LVLM hallucinations.

Problem

Hallucinations in large vision-language models are usually attributed to cross-modal attention imbalance; existing mitigations either ignore the spectral characteristics of visual information flow or rely on contrastive decoding, which doubles inference cost.

Approach

Identifies two hallucination patterns (Perceptual-Semantic Dissociation and Localized Fixation) from a frequency-domain view, scores attention heads with a Spectral Vortex Score (low-frequency concentration plus orientation isotropy of the attention map's log-magnitude spectrum), and modulates DCT coefficients of value matrices (boost high frequencies) and attention scores (boost low frequencies)

Key results

• On POPE-MS-COCO Random with LLaVA-1.5-7B, FLASH reaches 90.03% accuracy vs. 88.97% vanilla and 89.30% for PAI, while the authors report 1.69x inference time versus 1.87x for PAI and 1.98x for VCD.
• On LLaVA-1.5-13B POPE Popular, FLASH reaches 86.70% accuracy vs. 85.80% vanilla.
• On AMBER with LLaVA-1.5-7B, FLASH lowers Hal from 30.50 to 25.00 and Cog from 3.20 to 2.15, though its Cover (51.00) matches vanilla and is below VCD's 51.55.

Surprising findings

• Linear amplification of visual attention in the spatial domain gives little gain, and PAI's reported improvement is attributed mainly to contrastive decoding rather than attention manipulation.

Limitations

• Still relies on multiple spectral transforms (DCT/IDCT) and filtering operations, adding overhead.
• Applies value and attention modulation simultaneously without decoupling their effects.
• Unclear applicability to Q-Former-based models.

Experimental setup

Datasets: POPE (MS-COCO) MME CHAIR AMBER

Benchmarks: POPE MME CHAIR AMBER

Metrics: accuracy F1 MME score CHAIR_S CHAIR_I AMBER Cover AMBER Hal AMBER Cog

Baselines: VCD PAI SID

Builds on: Contrastive Decoding (VCD) PAI POPE CHAIR

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)
• Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory (ICML 2026 · similarity 0.92)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.92)
• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.91)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.91)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.91)