Research map / Vision-language and multimodal / Multimodal LLMs

Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies mask-prior drift and positional attention collapse as causes of repetition and poor visual grounding in large diffusion VLMs, and proposes two training-free inference-time fixes (Mask Prior Suppression and Monotonic RoPE Scaling) that improve LLaDA-V and LaViDa.

Problem

Large diffusion vision-language models (LDVLMs) suffer from repetitive generation and degraded visual grounding in long-form generation, especially with few decoding steps.

Approach

Mask Prior Suppression removes the component of final-layer mask-token hidden states aligned with a vocabulary-mean prior subspace, and Monotonic RoPE Scaling applies a sigmoid-gated, frequency-dependent amplification of low-frequency RoPE components so generation tokens keep attending to distant visual tokens; both require no training.

Key results

• On LLaDA-V, the method raises RefCOCOg from 64.8 to 65.0 and Ferret from 60.4 to 62.9, and DetailCaps from 59.8 to 63.6.
• On LaViDa, the method raises DetailCaps from 8.3 to 56.1 and RefCOCOg from 36.9 to 44.0 and Ferret from 25.9 to 35.7.
• On CHAIR (COCO), LLaDA-V CHAIR_s drops from 29.4 to 27.0 and CHAIR_i from 9.5 to 8.3.

Surprising findings

• The uncontextualized mask token has its highest logit on the end-of-text token in LaViDa, so large step counts cause premature termination and DetailCaps CAPTURE peaks at 16 steps.
• Standard RoPE scaling methods (NTK, YaRN) often degrade visual grounding on LLaDA-V rather than improve it.

Limitations

• Inference-time method whose effectiveness depends on the quality of the underlying pretrained representations.
• Evaluated only under full-suffix iterative unmasking; semi-autoregressive or block-wise diffusion decoding not studied.
• Vocabulary-mean prior is an empirical proxy; authors explicitly do not claim it fully characterizes mask-token collapse dynamics.

Experimental setup

Datasets: COCO 2014 validation (analysis)

Benchmarks: MME MMBench MMMU RefCOCOg Ferret GQA LLaVA-Bench DetailCaps MIA CHAIR AMBER-G

Metrics: accuracy CIDEr distinct-n repetition ratio CHAIR Cover HAL

Baselines: LLaDA-V LaViDa LLaVA-1.6-7B LLaVA-OneVision-7B Qwen2.5-VL-7B InternVL3-8B NTK-aware RoPE scaling YaRN DPad

Builds on: LLaDA-V LaViDa RoPE NTK-aware RoPE scaling YaRN

Closest papers by meaning

• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.92)
• Rethinking Causal Mask Attention for Vision-Language Inference (ICLR 2026 · similarity 0.91)
• Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
• First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.90)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
• A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models (ICLR 2026 · similarity 0.90)