Research map / Vision-language and multimodal / Cross-modal learning

When to Align, When to Predict: A Phase Diagram for Multimodal Learning

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Unimodal-to-multimodal representation transfer · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes cross-modal alignment (CA, via CCA/VICReg) versus cross-modal prediction (CP, via reduced-rank regression/reconstruction) under a spiked signal-plus-noise model with cross-modal nuisance correlation, deriving separation ratios that yield a four-regime phase diagram and a data-driven regime estimator validated on synthetic, stereo-vision, image-caption, astronomy, and single-cell data.

Problem

When cross-modal alignment versus cross-modal prediction succeeds or fails for multimodal representation learning, especially when nuisance structure is correlated across modalities, and when cross-modal training can beat the strongest single modality.

Approach

Under a spiked model with structured cross-modal nuisance correlation, CA whitens each view and its recovery is governed by a ratio of signal to cross-modal nuisance correlation, while CP whitens only the source and its recovery depends on source quality and target noise; the two separation ratios partition problems into Both, CA only, CP only, and Neither regimes, estimable from a small labeled s

Key results

• On synthetic spiked data, CP fails (subspace distance to 1) at nuisance correlation about 0.15 while CA holds near-perfect recovery until about 0.75.
• On LAMOST x Kepler (Both regime), CA and CP each win on different tasks; CA reaches R^2 0.620 on age vs 0.431 for LAMOST-only, while CP stays at the LAMOST ceiling for log g (0.976 vs 0.977).
• On LAMOST x TESS (Neither regime), no cross-modal method beats LAMOST-only on any target; on BMMC CITE-seq, CA reaches 0.606 balanced accuracy vs 0.584 RNA-only with partial recovery (r about 13 of 20).

Surprising findings

• At full nuisance alignment, CP's overcapacity bottleneck encodes both signal and nuisance without compression pressure, contrary to the linear theory's failure prediction.
• On astrophysical age prediction, the reverse CP direction (photometry to spectra) outperforms forward CP, reversing the preferred direction.

Limitations

• Theory is derived in the linear population setting; nonlinear validation relies on experiments rather than proofs.
• Regime estimation assumes two-stage pipelines where unimodal features approximately satisfy the spiked decomposition; single-stage pipelines give less reliable predictions.
• Downstream value of partially recovered subspaces is not determined by the separation ratios alone.

Experimental setup

Datasets: dSprites 3DShapes MS-COCO 2017 LAMOST DR8 Kepler DR25 TESS QLP BMMC CITE-seq

Benchmarks: Stereo-dSprites Stereo-3DShapes MS-COCO LAMOST x Kepler LAMOST x TESS BMMC CITE-seq

Metrics: linear probe accuracy top-1 accuracy R^2 balanced accuracy subspace distance separation ratio

Baselines: CA (VICReg) CP (MSE reconstruction) CP reverse direction RNA-only baseline LAMOST-only baseline photometry-only baseline DeepCCA

Builds on: VICReg DeepCCA CLIP masked autoencoders scGPT

Closest papers by meaning

• Understanding the Emergence of Multimodal Representation Alignment (ICML 2025 · similarity 0.90)
• On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.89)
• AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · similarity 0.89)
• A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity (NeurIPS 2025 · similarity 0.88)
• Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.88)
• A Closer Look at Multimodal Representation Collapse (ICML 2025 · similarity 0.88)