Research map / Optimization / Implicit bias and generalization
Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a one-hidden-layer leaky ReLU network with online SGD, without explicit regularization | Arbitrary-width network from arbitrary initialization · Adversarial label noise on sub-exponential distributions | classification error: tilde O(√optlin) | Helped |
| Use the SGD-trained network instead of the best linear classifier | One-hidden-layer leaky ReLU network · 𝒟_{b,0.5}; optlin ≤ 0.30 | test classification accuracy: almost exactly equal to 1-optlin | No clear effect |
| Use the SGD-trained network instead of the best linear classifier | One-hidden-layer leaky ReLU network · 𝒟_{b,0.5}; optlin > 0.30 | test classification accuracy: slightly outperforms the best linear classifier | Helped |
| Use the SGD-trained network instead of the best linear classifier | One-hidden-layer leaky ReLU network · 𝒟̃_b | test classification accuracy: performs significantly better | Helped |
| Replace leaky ReLU with tanh (ablation) | One-hidden-layer network on 𝒟_{b,0.5} · 𝒟_{b,0.5} | test classification accuracy: essentially the same | No clear effect |
| Vary the learning rate from η=0.01 to η ∈ {0.1, 0.001} (ablation) | One-hidden-layer leaky ReLU network on 𝒟_{b,0.5} · 𝒟_{b,0.5} | test classification accuracy: essentially the same | No clear effect |
| Change first-layer initialization variance from 1/m to m^{-2} or 1 (ablation) | One-hidden-layer leaky ReLU network on 𝒟_{b,0.5} · 𝒟_{b,0.5} | test classification accuracy: essentially the same | No clear effect |
| Use 100 epochs of multiple-pass SGD with batch size 32 rather than online SGD (ablation) | One-hidden-layer leaky ReLU network on 𝒟_{b,0.5} · 𝒟_{b,0.5} | test classification accuracy: virtually indistinguishable | No clear effect |
| Increase width from m=1,000 to m=100,000 (ablation) | One-hidden-layer leaky ReLU network on 𝒟_{b,0.5} · 𝒟_{b,0.5} | test classification accuracy: imperceptible | No clear effect |
| Add trainable biases and train second-layer weights, using initialization variance 1/m (ablation) | One-hidden-layer network on 𝒟_{b,0.5} · 𝒟_{b,0.5} | test classification accuracy: come close to Bayes-optimal classifier accuracy | Helped |
| Use initialization variance 1/m⁴ with trainable biases and second-layer weights instead of 1/m (ablation) | One-hidden-layer network on 𝒟_{b,0.5} · 𝒟_{b,0.5} | test classification accuracy: only learns a linear decision boundary | Hurt |
| Use a four-layer leaky ReLU network instead of a bias-free one-hidden-layer network (ablation) | SGD-trained network on 𝒟_{b,0.5} · 𝒟_{b,0.5} | test classification accuracy: generalize almost as well as the Bayes optimal classifier | Helped |
Closest papers by meaning
- SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data (ICLR 2018 · similarity 0.90)
- On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications (ICLR 2022 · similarity 0.88)
- Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data (ICLR 2024 · similarity 0.87)
- Benign overfitting in leaky ReLU networks with moderate input dimension (NeurIPS 2024 · similarity 0.87)
- On the Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views: Representation Complexity and Training Dynamics (ICML 2025 · similarity 0.87)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.