Research map / Vision-language and multimodal / Multimodal LLMs
Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diagnoses visual representation degradation in mid-layers of MLLM LLMs and proposes Predictive Regularization (PRe), a cosine-similarity loss forcing mid-layer visual tokens to predict the pre-LLM visual anchors, yielding gains on VLM benchmarks.
Problem
Next-token-prediction training of MLLMs may degrade the intrinsic visual representations inside the LLM, both globally (linear probe accuracy) and at the patch level (semantic separability), harming vision-centric capability.
Approach
Treats the degradation as a 'visual sacrifice' caused by language-only training and adds a lightweight predictive regularizer: a 2-layer MLP predicts the detached pre-LLM visual features from mid-layer LLM visual hidden states using negative cosine similarity, applied per patch and discarded at inference.
Key results
- With CLIP* + Vicuna-7B, PRe improves GQA 62.0→62.7, MMStar 30.3→34.6, and MMVP 20.0→22.0.
- With SigLIP2 (trainable) + Qwen2.5-7B, PRe raises RealWorldQA from 60.3 to 61.9 and TextVQA from 59.2 to 59.7.
- Applying PRe at the last LLM layer yields a drop versus baseline on several benchmarks, while mid-layer application gives the best results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PRe loss (predict degraded mid-layer visual features from pre-LLM anchors) | CLIP-ViT-L/14 (frozen) + Vicuna-7B-v1.5 · GQA, MMMU, AI2D, MMStar, TextVQA, MMVP | accuracy: +0.7 GQA, +0.4 MMMU, +1.7 AI2D, +4.3 MMStar, +1.1 TextVQA, +2.0 MMVP | Helped |
| add PRe loss | SigLIP2 (trainable) + Qwen2.5-7B-Instruct · RealWorldQA | accuracy: +1.6 (60.3 to 61.9) | Helped |
| add PRe loss | SigLIP2 (frozen) + Qwen2.5-7B-Instruct · MMMU, MMStar | accuracy: MMMU +0.3 / MMStar -0.1 for SigLIP2 frozen; MMStar -0.2 and SQA -0.4 for trainable SigLIP2 variant | Mixed |
| add PRe loss | CLIP (frozen) + Qwen2.5-7B-Instruct · GQA, MMMU, MMVP | accuracy: GQA -0.1, MMMU -0.7 for CLIP frozen; MMVP +3.4 for CLIP frozen | Mixed |
| apply PRe loss at last layer instead of mid-layer (ablation) | CLIP (frozen) + Vicuna-7B-v1.5 · GQA, MMMU, TextVQA, RWQA, MMVP | accuracy: Last-layer GQA 62.4 vs mid 62.7; MMVP 25.3 vs 22.0 (mid-layer on other benchmarks higher) | Mixed |
| use DINOv2 features as anchor instead of pre-LLM features (ablation) | CLIP (frozen) + Vicuna-7B-v1.5 · GQA, MMVP | accuracy: GQA 62.8 vs 62.7 (pre-LLM); MMVP 28.7 vs 22.0 (pre-LLM) | Mixed |
| use pre-projection encoder features as anchor (ablation) | CLIP (frozen) + Vicuna-7B-v1.5 · MMVP | accuracy: 32.7 vs 22.0 (pre-LLM) | Helped |
| apply PRe loss on globally aggregated features instead of patch-level features (ablation) | MLLM with PRe · general benchmarks | accuracy | Hurt |
| add PRe loss | CLIP-ViT-L/14 (frozen) + Qwen2.5-3B-Instruct · GQA, MMMU, AI2D, MMStar, SQA, TextVQA, OCRbench, RWQA, MMVP | accuracy: e.g. RWQA +2.8, MMVP -0.6, MMStar -0.2 | Mixed |
| add PRe loss | LLaVA-style models on additional benchmarks · MME-P, MMBench-EN, MMBench-CN, SEED-I, POPE | accuracy: MMB-en dropped 70.1 to 68.3 for clip+qwen2.5-3b; MMB-cn +2.0; POPE +1.2 acc | Mixed |
| increase lambda beyond default 0.5 (to 2.0) (ablation) | CLIP + Vicuna-7B-v1.5 PRe · MMMU, TextVQA, RealWorldQA | accuracy | Hurt |
Surprising findings
- Applying PRe to the last LLM layer can hurt performance, which the authors attribute to deep visual tokens being decoded to high-frequency inert tokens (e.g., '_in', '.').
- Using DINOv2 or DINOv3 as anchor features gives smaller gains than the internal pre-LLM anchor.
Limitations
- Predicting initial visual features may inherit biases of the original visual encoder.
- Contrastive and other SSL paradigms for MLLM pre-training are not explored.
- Gains are sometimes small or negative on some benchmarks (e.g., MMBench-en drop in one setting, some Qwen2.5-7B CLIP results).
Experimental setup
Datasets: LLaVA-558K LLaVA-665K COCO-Stuff CUB-200 ImageNet-100
Benchmarks: GQA MMMU AI2D MMStar ScienceQA-IMG TextVQA OCRBench RealWorldQA MMVP MME MMBench SEED-Bench-IMG
Metrics: accuracy
Baselines: LLaVA-1.5 LLaVA-NeXT DINOv2 anchor last-layer PRe
Builds on: LLaVA-1.5 SimSiam JEPA predictive coding
Closest papers by meaning
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.93)
- The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss (ICLR 2026 · similarity 0.92)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.91)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
- Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.