Research map / Learning theory and algorithms / Kernels and regression theory
The Breakdown of Gaussian Universality in Classification of High-dimensional Linear Factor Mixtures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a high-dimensional asymptotic characterization of ridge-regularized ERM classification on linear factor mixture models and shows Gaussian universality breaks down except under Gaussian informative factors or square loss.
Problem
Whether Gaussian equivalent principles (conditional Gaussian equivalence for classification of Gaussian mixtures) hold for classification on general non-Gaussian mixture data, and when the data distribution matters beyond class means and covariances.
Approach
Models classes as linear factor mixtures (LFMM) with independent non-Gaussian informative and noise factors, derives via a leave-one-out analysis a self-consistent system of equations for ridge-regularized ERM performance, and shows Gaussian universality holds only if informative factors are Gaussian or the loss is square loss.
Key results
- Gaussian universality of in-distribution performance holds iff the informative-factor noises e_1..e_q are Gaussian, regardless of noise factor distributions.
- Classifier Gaussian universality holds iff informative factors are Gaussian or the loss derivative is linear in the score (square loss only).
- For LFMM with non-Gaussian informative factors, logistic loss outperforms square loss at optimally tuned ridge regularization, while under the equivalent GMM it does not.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use logistic loss instead of square loss with optimally chosen regularization | ERM classifier on LFMM with non-Gaussian informative factors · LFMM test distribution | classification accuracy | Helped |
| Use logistic loss instead of square loss | ERM classifier on equivalent GMM · GMM test distribution | classification accuracy | No clear effect |
| Use square loss rather than a non-square loss | ERM classifier trained on LFMM · LFMM test distribution | predictive ability | Hurt |
| Replace non-Gaussian informative factors with Gaussian factors while preserving class means and covariances | ERM training and evaluation on LFMM · In-distribution performance | classification accuracy | No clear effect |
| Replace non-Gaussian informative factors with Gaussian factors while preserving class means and covariances | ERM classifier trained on LFMM · Arbitrary test distribution | classification error | No clear effect |
| Use square loss, whose derivative is linear in the score | ERM classifier trained on LFMM · LFMM test distribution | classification error | No clear effect |
Surprising findings
- Square loss is no longer optimal for ridge-regularized classification once Gaussian universality breaks down on non-Gaussian mixtures.
- Gaussian universality can break even for data with concentrated quadratic forms, a condition elliptical data violates.
Limitations
- Assumes smooth convex losses; non-smooth losses like hinge loss are excluded.
- Analysis restricted to binary classification; multi-class extension is stated as straightforward but not done.
- Assumes fixed number of informative factors and independent factors.
Experimental setup
Datasets: synthetic linear factor mixture data
Metrics: classification accuracy histograms of predicted scores vs theoretical densities
Baselines: square loss ERM logistic loss ERM square hinge loss ERM
Builds on: conditional Gaussian equivalent principle Gaussian mixture classification analyses leave-one-out analysis for robust regression
Closest papers by meaning
- Minimax Optimal Early-Stopped Gradient Descent for Gaussian Mixture Classification (NeurIPS 2026 · similarity 0.86)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.86)
- Learning Mixtures of Gaussian Processes through Random Projection (ICML 2024 · similarity 0.86)
- Universality in Transfer Learning for Linear Models (NeurIPS 2024 · similarity 0.85)
- Toward Global Convergence of Gradient EM for Over-Paramterized Gaussian Mixture Models (NeurIPS 2024 · similarity 0.84)
- Out-Of-Domain Unlabeled Data Improves Generalization (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.