Research map / Learning theory and algorithms / Prediction and decision losses
A Global Geometric Analysis of Maximal Coding Rate Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes all critical points of the regularized MCR^2 objective in the unconstrained feature model, showing every critical point is either a local maximizer (with orthogonal low-dimensional class subspaces) or a strict saddle, and validates with synthetic GD and MLP experiments on MNIST/CIFAR-10.
Problem
Understanding the theoretical optimality and optimization landscape of the maximal coding rate reduction (MCR^2) objective, whose non-concave log-determinant structure has been unanalyzed beyond global optima.
Approach
Derives closed-form characterizations of all local and global maximizers of the regularized MCR^2 objective in the unconstrained feature model and proves every other critical point is a strict saddle, so first-order methods with random initialization converge to useful local maxima.
Key results
- On MNIST and CIFAR-10 subsets, the function value from training an MLP with Adam matches the closed-form optimum F* to two decimals (e.g., MNIST (1000,4): 37.38 vs 37.38).
- Synthetic experiment with d=100, K=4 recovers the predicted singular value structure; per-class nonzero singular value counts 24, 23, 27, 26 sum to 100.
- GD with random initialization converges linearly to the optimal value across tested synthetic settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train an MLP with ReLU using the regularized MCR² objective | MLP feature representations · MNIST subsets (1000, 4), (1500, 6), (2000, 8), and (2500, 10) | function value: 37.38 -> 37.38; 38.96 -> 38.96; 38.48 -> 38.48; 37.41 -> 37.41 | Helped |
| Train an MLP with ReLU using the regularized MCR² objective | MLP feature representations · CIFAR-10 subsets (1000, 4), (1500, 6), and (2000, 8) | function value: 215.61 -> 215.61; 229.14 -> 229.14; 230.70 -> 230.70 | Helped |
| Train an MLP with ReLU using the regularized MCR² objective | MLP feature representations · CIFAR-10 subset (2500, 10) | function value: 228.49 -> 228.48 | Hurt |
| Train an MLP with ReLU using the regularized MCR² objective | MLP feature representations · MNIST and CIFAR-10 subsets | class-subspace cosine similarity s: MNIST: $ 5.9 \cdot 10^{-6}$, $ 3.8 \cdot 10^{-6}$, 0.011, 0.008; CIFAR-10: 0.004, 0.029, 0.059, 0.171 | Helped |
Surprising findings
- In unbalanced class settings, global maximizers can collapse the largest classes to zero rank, so global optima may not be the best representations; equitable local optima may be preferred.
Limitations
- Analysis is in the unconstrained feature model and does not cover deep network parameterizations.
- Global maximizers may be undesirable under unbalanced class sizes.
- Experiments on real data are small subsets with a simple MLP.
Experimental setup
Datasets: MNIST CIFAR-10 synthetic data
Benchmarks: MNIST subsets CIFAR-10 subsets
Metrics: regularized MCR^2 function value gap class subspace discrimination s (max principal-angle cosine)
Builds on: maximal coding rate reduction (MCR^2) ReduNet unconstrained feature model strict saddle theory
Closest papers by meaning
- Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation (ICLR 2025 · similarity 0.86)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.86)
- An In-depth Investigation of Sparse Rate Reduction in Transformer-like Models (NeurIPS 2024 · similarity 0.86)
- Flat Channels to Infinity in Neural Loss Landscapes (NeurIPS 2025 · similarity 0.85)
- Layerwise LQR for Geometry-Aware Optimization of Deep Networks (NeurIPS 2026 · similarity 0.85)
- Block Coordinate Descent for Neural Networks Provably Finds Global Minima (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.