Research map / Vision-language and multimodal / Cross-modal learning

The Alignment Illusion in Multimodal Large Language Models

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that visual-text principal-angle and other scalar alignment scores in MLLMs stay high when visual tokens are replaced with Gaussian noise, traces this to shared MLP down-projection anisotropy, and proposes the principal-angle gap (PA gap) as a diagnostic that tracks task accuracy better.

Problem

Layer-wise visual-text similarity in MLLMs is read as evidence of content-level cross-modal integration, but scalar alignment scores may reflect shared language-model weights rather than visual content.

Approach

Visual-text alignment can be inflated by weight-induced alignment from anisotropic MLP down-projections shared across modalities; the gap between the top two principal-angle cosines (PA gap) isolates multi-directional visual structure from this one-directional component.

Key results

• Under Gaussian noise replacement of visual tokens, task accuracy drops by 38 to 50 pp across 13 MLLMs while sigma_1 changes little and medians of separation scores for sigma_1, CKA and SVCCA are negative.
• The PA gap achieves mean |r| = 0.894 (median 0.917) with accuracy under alpha-interpolation and has the expected sign in all 13 models, outperforming scalar baselines.
• Irrelevant images reduce accuracy below the text-only baseline by a median of 5.0 pp, and the PA gap orders Orig < Irr < Noise (0.130, 0.213, 0.324) while accuracy does not follow that order.

Surprising findings

• Scalar alignment scores (sigma_1, CKA, SVCCA, MIR) fail to separate original from Gaussian-noise visual tokens, even though accuracy collapses by 38-50 pp.
• Irrelevant images reduce accuracy below text-only performance, indicating the model actively processes unrelated visual content rather than ignoring it.

Limitations

• Analysis limited to projector-LLM architectures and controlled multiple-choice evaluation on MMBench.
• Projector-output interventions only; open-ended generation, video, document and agentic tasks not tested.
• The PA gap's ability to rank the irrelevant-image regime does not translate into accuracy ordering, so it cannot establish usefulness of visual content for the question.

Experimental setup

Datasets: MMBench

Benchmarks: MMBench

Metrics: accuracy Pearson correlation |r| separation score McNemar test principal-angle cosine CKA SVCCA MIR

Baselines: CKA SVCCA MIR sigma_1 (leading principal-angle cosine) participation ratio spectral entropy

Builds on: CKA SVCCA principal angles LLaVA-OneVision Qwen2.5-VL

Closest papers by meaning

• Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.91)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.91)
• The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss (ICLR 2026 · similarity 0.90)
• Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions (ICML 2026 · similarity 0.90)
• When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.89)