Research map / Architectures and efficiency / RNNs and normalization
Feature Learning in $L_2$-regularized DNNs: Attraction/Repulsion and Sparsity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reformulate the regularized objective using hidden representations | L2-regularized DNNs · Training-set representations across layers | locality of the representation objective | Helped |
| Use positively homogeneous activations and express the objective through covariance pairs | L2-regularized DNNs · Any training set of size N, with hidden widths at least N(N+1) | dependence of the optimum on hidden width: N(N+1) | Helped |
| Increase hidden-layer width | L2-regularized DNNs with positively homogeneous activations · Training objective, fixed depth | minimum regularized loss: N(N+1) | No clear effect |
| Use L2 regularization in the MNIST experiment | Fully connected DNN · MNIST, N=1000 | width at which loss plateau begins: roughly at a width of 10 | No clear effect |
| Use a constructed ReLU-network dataset with high-rank optimal hidden covariances | Shallow L2-regularized ReLU DNN · Constructed dataset, N datapoints | hidden representation rank required at an optimum: N^{2}/4 | Hurt |
| Use one-hot classification outputs | Shallow ReLU network · Classification with n_L classes | hidden covariance rank: n_L | Helped |
| Use scalar inputs and outputs | Shallow ReLU network · Scalar-input/output task with N datapoints | width needed for a width-independent optimum: 4N | Helped |
| Optimize the representation reformulations directly rather than the weights | L2-regularized DNN training · Numerical optimization | computational efficiency and stability | Hurt |
Closest papers by meaning
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.88)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.87)
- Asymptotics of feature learning in two-layer networks after one gradient-step (ICML 2024 · similarity 0.87)
- Exploring The Loss Landscape Of Regularized Neural Networks Via Convex Duality (ICLR 2025 · similarity 0.87)
- How does Gradient Descent Learn Features --- A Local Analysis for Regularized Two-Layer Neural Networks (NeurIPS 2024 · similarity 0.87)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.