Research map / Optimization / Implicit bias and generalization
Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Input
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies mean-field SGD for two-layer networks learning subspace-sparse polynomials under Gaussian input, giving a basis-free necessary condition (reflective property) and a nearly matching sufficient condition with dimension-free exponential loss decay.
Problem
Characterizing when SGD on two-layer neural networks can learn a polynomial target that depends only on the projection of a standard Gaussian input onto a low-dimensional unknown subspace, in a basis-free (rotation-invariant) setting.
Approach
Generalizes the merged-staircase property to a basis-free "reflective property" w.r.t. a subspace S, showing that if it holds the mean-field flow stays in w_S=0 and cannot learn h* on S; for the sufficiency side, a two-stage training with averaging of p independent trajectories gives algebraic independence and dimension-free exponential decay.
Key results
- Necessary condition: if the reflective property holds for a nontrivial subspace S, then inf over t in [0,T] of the loss is at least (1/2)E[|h*(z)-h*_{S^perp}(z_S^perp)|^2] - C/d^{1/2}, bounded away from zero for large d.
- Sufficient condition (under a Taylor-expansion non-trapping assumption): with the proposed two-stage algorithm, the loss decays as C1 exp(-C2 t) with dimension-free constants.
- Proposition: the kernel smallest eigenvalue satisfies lambda_min(K(t)) >= C t^{2sn binom(n+p,p)}, and the reflective property with nontrivial S is shown equivalent to isoLeap(h*) >= 2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed two-stage training strategy under the Taylor-expansion condition | Two-layer mean-field network · Subspace-sparse polynomials with Gaussian input | loss: \mathcal{E}(\rho_t) \leq C_1 \exp(-C_2 t),\quad\forall~t\geq 0 | Helped |
| Have the target satisfy the reflective property with respect to a nontrivial subspace S | One-pass SGD mean-field flow · Subspace-sparse polynomials with Gaussian input | loss: \inf_{0\leq t\leq T}\mathcal{E}(\rho_t)\geq \frac{1}{4}\mathbb{E}_{z\sim \mathcal{N}(0,I_V)}\left[|h^*(z) - h_{S^\perp}^ | Hurt |
Limitations
- Gap between the necessary condition (reflective property) and the sufficient condition, which relies on Taylor expansion of the flow.
- Training strategy averages p independent trajectories and uses a modified polynomial activation (1+zeta)^n, which departs from standard training.
Experimental setup
Baselines: merged-staircase property isoLeap NTK regime
Builds on: merged-staircase property leap complexity mean-field analysis of two-layer networks
Closest papers by meaning
- Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit (NeurIPS 2024 · similarity 0.88)
- Learning Hierarchical Polynomials with Three-Layer Neural Networks (ICLR 2024 · similarity 0.84)
- Learning Polynomial Problems with $SL(2, \mathbb{R})$-Equivariance (ICLR 2024 · similarity 0.84)
- The Generative Leap: Tight Sample Complexity for Efficiently Learning Gaussian Multi-Index Models (NeurIPS 2025 · similarity 0.83)
- Statistical Guarantees for High-Dimensional Stochastic Gradient Descent (NeurIPS 2025 · similarity 0.83)
- Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.