Research map / Vision-language and multimodal / Multimodal LLMs

Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LaVer, a training framework that adds masked image modeling in the LLM's latent space (EMA teacher, clipped Gram-anchoring) to counter visual token homogenization in MLLMs, improving benchmark results especially on dense OCR and vision-centric tasks.

Problem

MLLMs suffer modality imbalance: visual representations homogenize and are underused in deeper layers because next-text-token prediction gives only indirect visual supervision, causing degraded visual performance and hallucinations.

Approach

Randomly mask vision tokens in the input embedding space and train the MLLM to predict an EMA teacher's latent visual logits for masked positions (MIM in the LLM's joint latent space), with a one-sided Clipped Gram-Anchoring regularizer to prevent collapse to identical visual features, plus mixed bidirectional attention for vision tokens.

Key results

• With SigLIP 2, LaVer improves OCRBench from 536 to 639 (+103 points, 19.22%) and average score from 55.72 to 57.87.
• With CLIP, LaVer improves ChartQA by 6.07% and MMVP by 12.00%, with average gain of 2.66 points (50.58 to 53.24).
• Against ROSS under identical settings, LaVer gains 2.91% (CLIP) and 1.78% (SigLIP) on average versus ROSS's 2.25% and 1.45%.

Surprising findings

• Plain MIM alone degrades performance (SigLIP 2 average 55.72 to 53.76) and increases vision-token cosine similarity, revealing a visual feature inconsistency failure mode.
• Mixed attention alone captures part of the gain but 2D-RoPE alone has minimal impact (55.57 vs 55.72 average for SigLIP 2).

Limitations

• Adds training overhead: 13-16% longer training time and 14-26% more memory in stage 2.
• Reported gains are modest on many general benchmarks, and some benchmarks show regressions for particular encoders.
• Higher mask ratios under cosine scheduling degrade performance, implying sensitivity to masking schedule.

Experimental setup

Datasets: LLaVA-558K FineVision LLaVA-OneVision 4M MMVP LLaVA-665K

Benchmarks: GQA MMBench SEED-Image MME RealWorldQA MMMU MMStar OCRBench TextVQA ChartQA AI2D CV-Bench-2D

Metrics: accuracy score gIoU

Baselines: ROSS LLaVA-OneVision 1.5 recipe baseline (next-token prediction only)

Builds on: iBOT Gram-Anchoring (DINOv3) LLaVA-OneVision 1.5 ROSS

Closest papers by meaning

• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.93)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)
• UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.91)
• The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss (ICLR 2026 · similarity 0.91)
• Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.91)