Research map / Label-efficient and robust learning / Recognition and long-tail
Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies model collapse in DCCA-based multi-view representation learning and proposes NR-DCCA, which adds a noise-regularization loss enforcing a correlation-invariant property to keep weight matrices full-rank, plus a synthetic benchmark with controllable common rate.
Problem
DCCA-based multi-view representation learning methods perform well early in training but their downstream performance collapses as training continues, making early stopping hard to decide.
Approach
Adds a noise regularization loss that forces the DNN transformation to preserve the correlation between real data and Gaussian noise (Correlation Invariant Property), which the paper proves is equivalent to the weight matrix being full-rank, thereby preventing low-rank, redundant weights and model collapse.
Key results
- NR-DCCA avoids the model collapse of DCCA across synthetic common-rate datasets, achieving state-of-the-art and stable performance versus DCCA-based baselines.
- On real-world datasets (PolyMnist, CUB, Caltech), NR-DCCA shows competitive and stable performance while several DCCA variants collapse.
- Theory proves CIP of a square weight matrix holds if and only if it is full-rank; the noise regularization also generalizes to DGCCA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add noise regularization (NR) | DCCA · Synthetic datasets, all common rates | R2 | Helped |
| Add noise regularization (NR) | DCCA · PolyMnist, CUB, Caltech | average F1 score | Helped |
| Continue training without noise regularization | DCCA-based methods · Synthetic datasets, all common rates | R2 | Hurt |
| Increase the depth of the MLP (ablation) | DCCA · Synthetic and real-world datasets | training stability | Hurt |
| Increase the ridge regularization parameter (ablation) | DCCA · CUB | downstream performance | Hurt |
| Increase the NR weight α too much (ablation) | NR-DCCA · CUB | training convergence | Hurt |
| Decrease the NR weight α too much (ablation) | NR-DCCA · CUB | model collapse | Hurt |
| Use uniformly distributed noise instead of Gaussian white noise (ablation) | NR-DCCA · NR-DCCA noise regularization | model collapse prevention | No clear effect |
Surprising findings
- Ridge regularization, commonly used in DCCA, does not prevent low-rank weights and can even worsen performance and increase noise correlation.
- In DCCA-based methods, correlation between unrelated Gaussian noise and data increases during training, while NR-DCCA keeps it invariant.
Limitations
- Model collapse is only demonstrated empirically and theoretically for simple, shallow cases; the full-rank argument is for square weight matrices.
- Real-world DCCA collapse is less uniform and the paper notes the need for per-dataset alpha tuning (1.5, 2, 10 for CUB, PolyMnist, Caltech).
- Synthetic benchmark is constructed by the authors and not released in the text.
Experimental setup
Datasets: PolyMnist CUB Caltech101 synthetic God-embedding datasets
Benchmarks: PolyMnist CUB Caltech101 synthetic common-rate datasets
Metrics: R2 F1 score reconstruction loss denoising loss NESum
Baselines: CONCAT PRCCA KCCA Linear CCA DCCA DGCCA DCCA_EY DCCA_GHA DCCAE DCCA_PRIVATE MVTCAE
Builds on: Deep CCA Linear CCA DGCCA DCCAE CCA-Zoo
Closest papers by meaning
- On Sparse Canonical Correlation Analysis (NeurIPS 2024 · similarity 0.83)
- Unconstrained Stochastic CCA: Unifying Multiview and Self-Supervised Learning (ICLR 2024 · similarity 0.82)
- A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective (NeurIPS 2025 · similarity 0.82)
- MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling (CVPR 2025 · similarity 0.82)
- Deep Incomplete Multi-view Learning via Cyclic Permutation of VAEs (ICLR 2025 · similarity 0.81)
- MaxSup: Overcoming Representation Collapse in Label Smoothing (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.