Research map / Vision-language and multimodal / Multimodal LLMs

The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

The paper argues that Pre-Norm causes a norm disparity between visual and text tokens in MLLMs, creating "representational inertia" for visual tokens, and proposes inserting a single initialized LayerNorm after the visual projector with a Global Weight Compensation gradient hook, reporting gains on LLaVA-1.5-style models including MMLU.

Problem

Visual tokens in Pre-Norm MLLMs have much larger L2 norms than text tokens, which the authors claim makes visual representations update too slowly and impairs cross-modal fusion.

Approach

Theoretically frames norm disparity as causing asymmetric angular update rates, and fixes it with a LayerNorm after the projector whose gain is initialized to the text-embedding norm, plus a backward gradient compensation (GWC) to avoid vanishing gradients to the vision encoder.

Key results

• On Llama-3.2-3B, adding the norm layer with GWC raises average over nine benchmarks from 59.01 (w/o Norm) to 61.27, and naive norm alignment without GWC reaches 62.04.
• On Qwen2.5-7B, norm alignment with GWC raises the average from 68.49 to 69.41, with MMLU improving from 71.02 to 71.74.
• Ablation shows default-initialized LayerNorm gain barely changes after Stage 1 (L2 norm 53.25) while the proposed init changes the gain meaningfully (L2 norm 2.28).

Surprising findings

• Naive norm alignment without GWC gives the best average on Llama (62.04 vs 61.27 with GWC), contradicting the claimed need for GWC there.
• Text-only gains on MMLU (e.g., 45.19 to 53.21 on Llama) are large despite the intervention targeting visual tokens.

Limitations

• Theoretical analysis relies on strong simplifying assumptions (uniform update magnitude, consistent update geometry, random rotation direction).
• Gains are mixed: GWC is not uniformly better (e.g., Llama averages favor no-GWC variant), and the authors note possible gradient oscillation.
• Evaluation limited to LLaVA-1.5-style setup with SigLIP and two LLM backbones; no high-resolution setting.

Experimental setup

Datasets: LLaVA-558K LLaVA-NeXT MMBench POPE MM-Star

Benchmarks: MMBench-EN MM-Star POPE SEED-Bench-2-Plus OCRBench ScienceQA AI2D HellaSwag MMLU

Metrics: accuracy

Baselines: w/o Norm (LLaVA-1.5 style baseline) w/ Norm (w/o GWC)

Builds on: LLaVA-1.5 Pre-Norm Transformer LayerNorm RMSNorm

Closest papers by meaning

• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.92)
• Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• The Alignment Illusion in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
• Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.90)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.90)