Research map / Architectures and efficiency / RNNs and normalization
Low Rank Gradients and Where to Find Them
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis showing that the input-weight gradient of two-layer networks under anisotropic spiked data is approximately rank two (residue spike plus data spike linked by an interpolant), with characterization of how scaling, activation, and regularizers (weight decay, input noise, Jacobian penalty) shift the balance.
Problem
Low-rank structure of the first-layer gradient in two-layer networks has been studied only under idealized isotropic or well-conditioned data and weights; the effects of anisotropy, ill-conditioning, scaling regime, activation, and regularization are not characterized.
Approach
Decompose the one-step gradient into a residue-aligned rank-one term S1, a data-spike rank-one term S2 and an interpolant S12 under a spiked anisotropic covariance model; prove the gradient is approximately rank one or rank two depending on spike size, and show how activations, MF vs NTK scaling, and regularizers modulate the components.
Key results
- Theory: for spike exponent nu below 1/4 the gradient is approximately rank one aligned with S1+S12; for nu in [1/4,1/2) it becomes approximately rank two S1+S12+S2 under C2 activations.
- ReLU suppresses the residue spike S1 relative to smooth activations (ELU, Swish, Softplus), and small input noise (tau^2=0.25) re-emerges S1.
- Under MF scaling the dominant gradient direction stays stable during early training, whereas under NTK scaling it evolves significantly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReLU instead of a smooth activation | Two-layer network gradient · Synthetic spiked data, ν = 1/8 | Residue-spike contribution S1 | Hurt |
| Add isotropic Gaussian input noise | Two-layer network gradient · Synthetic spiked data, ReLU, ν = 1/8 | Residue-spike contribution S1: τ^2=0.25 | Helped |
| Add isotropic Gaussian input noise | Two-layer network gradient · Synthetic spiked data, ReLU, ν = 1/8 | Data-spike contribution S2 relative to S1 and bulk: τ^2=0.25 | Hurt |
| Add isotropic Gaussian input noise | Two-layer network gradient on MNIST · MNIST | Residue-spike contribution S1: σ²=100 | Helped |
| Add isotropic Gaussian input noise | Two-layer network gradient on MNIST · MNIST | Data-aligned spike: σ²=100 | Hurt |
| Increase Jacobian-penalty strength | Two-layer network gradient · Synthetic spiked data, NTK, ν = 3/8 | Residue-spike contribution S1: λ=0; λ=100 | Hurt |
| Increase Jacobian-penalty strength | Two-layer network gradient · Synthetic spiked data, NTK, ν = 3/8 | Data-spike contribution: λ=0; λ=100 | Helped |
| Apply Jacobian regularization | Two-layer network gradient on CIFAR-10 embeddings · CIFAR-10 embeddings | Residue-spike contribution S1: λ=10^5 | Hurt |
| Apply Jacobian regularization | Two-layer network gradient on CIFAR-10 embeddings · CIFAR-10 embeddings | Data-aligned spike: λ=10^5 | Helped |
| Increase weight decay until its gradient dominates the spike terms | Two-layer network gradient · Spiked data | Residue- and data-spike contributions: λ||W||₂ = ω(√m γ_m n^ν) | Hurt |
| Use MF rather than NTK scaling | Two-layer network training · Isotropic data, ReLU, MSE | Stability of the leading gradient direction during training | Helped |
| Use NTK rather than MF scaling | Two-layer network training · Single-index sigmoid targets, isotropic data | Alignment of the leading gradient direction with X_B^T r | Helped |
Surprising findings
- Under MF scaling the gradient direction stays stable during early training while under NTK it oscillates
- Phase transition in gradient alignment from residue to data spike during training is not visible in the training loss
Limitations
- Results are asymptotic with probability 1-o(1) and rely on proportional scaling and Gaussian data assumptions
- ReLU is not C2, so analysis requires separate treatment
- Phase transition during training is only described empirically and left for future work
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: singular value spectrum normalized inner product alignment mean principal angle
Builds on: spiked covariance model weight normalization single-index model analysis
Closest papers by meaning
- Generalization Bounds for Rank-sparse Neural Networks (NeurIPS 2025 · similarity 0.88)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.87)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.87)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.87)
- A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks (ICML 2024 · similarity 0.86)
- Weight decay induces low-rank attention layers (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.