AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Limitations of Lazy Training of Two-layers Neural Network

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, Andrea Montanari

NeurIPS 2019 · Architectures and efficiency › RNNs and normalization · approach: Theory of depth and width · 49 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Train all network weights rather than use the neural tangent linearizationTwo-layer network with quadratic activation · Quadratic functions (Gaussian features), N/d in (0,1)prediction risk: R_{{\sf NN}, N}(f_*) \le R_{{\sf NT},N}(f_*)-{ m GAP}(\rho)Improved
Train all network weights to select principal eigendirectionsTwo-layer network with quadratic activation · Quadratic functions (Gaussian features), N>dprediction risk: zero riskImproved
Use random features with fixed first-layer weightsTwo-layer network · Quadratic functions (Gaussian features)prediction risk: remains generally bounded away from zeroHurt
Use random features with weights not aligned to the target quadraticRandom features · Quadratic functions (Gaussian features), ρ→∞normalized prediction risk: 1-\frac{\langle {\boldsymbol \Gamma},{\boldsymbol B}\rangle^2}{\|{\boldsymbol \Gamma}\|_F^2\|{\boldsymbol B}\|_F^2}Hurt
Increase the number of NT neurons to at least the feature dimensionNeural tangent model · Quadratic functions (Gaussian features)prediction risk: vanishes asymptotically for ρ ≥ 1Improved
Train all network weights rather than use RF or NTTwo-layer network · Mixture of Gaussiansprediction risk: strictly smallerImproved
Increase the number of neurons in the mixture-of-Gaussians settingNeural tangent model and fully trained network · Mixture of Gaussiansprediction risk: Bayes error is not achieved even for very wide networks N/d\gg 1No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy