Research map / Vision-language and multimodal / Cross-modal learning
Understanding Unimodal Bias in Multimodal Deep Linear Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes unimodal bias in multimodal deep linear networks, deriving how fusion depth, dataset correlation statistics, and initialization set the length of the unimodal learning phase, and checks predictions on ReLU networks and noisy MNIST.
Problem
Joint training of multimodal networks often leads to unimodal bias, where one modality is learned first and the other is ignored for a long period; theory of how architecture (fusion depth), data statistics, and initialization affect this bias is lacking.
Approach
Models multimodal fusion as deep linear networks with separate pre-fusion branches per modality, derives gradient flow fixed points and saddles, and computes the time ratio between learning the two modalities in closed form to leading order in initialization, showing deeper fusion prolongs the unimodal phase.
Key results
- Time ratio t_B/t_A = 1 + (||Sigma_yA|| - ||Sigma_yB||)/||Sigma_yB - Sigma_yA Sigma_A^-1 Sigma_AB||; the unimodal phase lengthens with fusion depth L_f, input-output correlation ratio, and inter-modality correlation.
- On noisy MNIST with MLPs, the time ratio rises from 1.30 at L_f=1 to 6.38 at L_f=5; with CNNs from 1.04 to 2.53.
- In overparameterized late fusion networks, long unimodal phases cause overfitting of the first modality and a permanent unimodal bias / generalization deficit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move the fusion layer deeper | five-layer multimodal ReLU MLP · Noisy MNIST | time ratio: 1.30±0.24 -> 6.38±1.12 | Mixed |
| move the fusion layer deeper | five-layer multimodal CNN · Noisy MNIST | time ratio: 1.04±0.03 -> 2.53±0.45 | Helped |
| use a deeper fusion layer | multimodal deep linear network · linear multimodal learning | duration of unimodal phase | Hurt |
| increase correlations between input modalities | multimodal deep linear network · linear multimodal learning | duration of unimodal phase | Hurt |
| increase disparity in input-output correlations between modalities | late fusion linear network · linear multimodal learning | duration of unimodal phase | Hurt |
| use a shallower fusion layer | intermediate fusion linear network · linear multimodal learning | duration of unimodal phase | Helped |
| increase initialization scale | intermediate fusion linear network · linear multimodal learning | duration of unimodal phase | Hurt |
| use early fusion instead of late fusion | multimodal linear network · underparameterized linear task | generalization error | No clear effect |
| use late fusion instead of early fusion | multimodal ReLU network · heterogeneous task y=x_A+XOR(x_B) | task learning success | Mixed |
Surprising findings
- Duration of the unimodal phase in late fusion depends only on dataset statistics, not network depth or initialization, since depth slows both modalities equally.
- Late fusion networks prioritize the modality more correlated with the output even when the other modality contributes more to the output (superficial modality preference).
Limitations
- Theory derived for linear networks and small initialization; time ratio is a leading-order approximation.
- Nonlinear extensions validated only in limited settings (linear target with ReLU, one noisy MNIST task).
Experimental setup
Datasets: synthetic Gaussian multimodal linear regression data noisy MNIST
Benchmarks: noisy MNIST
Metrics: time ratio of unimodal accuracy reaching 50% training loss generalization error time ratio t_B/t_A amount of mis-attribution
Baselines: early fusion linear network unimodal network
Builds on: Saxe et al. deep linear network dynamics modality competition theory (Huang et al.)
Closest papers by meaning
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.88)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.86)
- Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion (CVPR 2024 · similarity 0.86)
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance (ICML 2024 · similarity 0.85)
- Revealing Vision-Language Integration in the Brain with Multimodal Networks (ICML 2024 · similarity 0.85)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.