Research map / Optimization / Optimizers
Unconstrained Stochastic CCA: Unifying Multiview and Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an unconstrained, Eckhart-Young-style objective for generalized eigenvalue problems that enables stochastic gradient training of linear CCA/PLS, Deep CCA, and SSL (Barlow Twins/VICReg-like) methods, and proves linear Barlow Twins/VICReg recover CCA subspaces.
Problem
Linear CCA is computationally infeasible at large scale, and existing stochastic Deep CCA training is biased or complex; the relationship between CCA and joint-embedding SSL methods (Barlow Twins, VICReg) is poorly understood.
Approach
An unconstrained trace objective, derived from an Eckhart-Young-type inequality, whose maximiser spans the top-k GEP subspace; this allows unbiased mini-batch gradient estimates and out-of-the-box SGD for CCA, PLS, Deep CCA and SSL.
Key results
- Linear Barlow Twins and VICReg with untied linear weights recover a top-CCA subspace (proved, informal).
- The GEP-EY objective has no spurious local minima under an eigengap condition (informal proposition).
- Stochastic CCA method reportedly converges faster and reaches higher correlation than SGHA and gamma-Eigengame, though the excerpt gives no numbers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed stochastic CCA method instead of existing stochastic CCA techniques | stochastic CCA · CCA benchmarks (not specified) | convergence speed | Helped |
| use the proposed stochastic CCA method instead of existing stochastic CCA techniques | stochastic CCA · CCA benchmarks (not specified) | correlation | Helped |
| use unbiased stochastic gradients from DCCA-EY instead of prior Deep CCA methods | Deep CCA · Deep CCA validation | validation correlation | Helped |
| use DCCA-EY with smaller mini-batches | Deep CCA · memory-constrained applications | minimum usable mini-batch size | Helped |
| use the proposed SSL method instead of VICReg or Barlow Twins | SSL · SSL comparison (not specified) | performance | No clear effect |
| use the proposed SSL method | SSL · SSL hyperparameter settings (not specified) | robustness to other hyperparameters | Helped |
Surprising findings
- Linear Barlow Twins and VICReg, with no explicit CCA objective, provably recover CCA subspaces.
Limitations
- Global guarantees rely on an eigengap condition that the authors believe is an artifact of the proof.
- Several results (Barlow Twins optimality of symmetric weights, strict monotonicity) are only conjectured in remarks.
- The excerpt is a fragmentary draft with many unresolved references and placeholders.
Experimental setup
Datasets: UK Biobank
Benchmarks: UK Biobank
Metrics: canonical correlation validation correlation
Baselines: SGHA gamma-Eigengame Barlow Twins VICReg DCCA
Builds on: Deep CCA Barlow Twins VICReg EigenGame subspace Generalized Hebbian Algorithm
Closest papers by meaning
- Efficient Globally Convergent Stochastic Optimization for Canonical Correlation Analysis (NeurIPS 2016 · similarity 0.90)
- Nonparametric Canonical Correlation Analysis (ICML 2016 · similarity 0.89)
- On Sparse Canonical Correlation Analysis (NeurIPS 2024 · similarity 0.89)
- Unpaired Canonical Correlation Analysis (NeurIPS 2026 · similarity 0.89)
- Stochastic Approximation for Canonical Correlation Analysis (NeurIPS 2017 · similarity 0.89)
- Rethinking PCA Through Duality (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.