Research map / Optimization / Implicit bias and generalization
Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add singular-value truncation to the convex formulation (ablation) | Two-layer ReLU CNN with BN · CIFAR-10 | test accuracy: 66.91 -> 67.11 | Helped |
| Add singular-value truncation to the baseline CNN (ablation) | Two-layer ReLU CNN with BN · CIFAR-10 | test accuracy: 67.06 -> 66.62 | Hurt |
| Use a closed-form solution instead of GD | Two-layer ReLU network with BN · CIFAR-100, three-class subset | test accuracy: 60.1% -> 47.7% | Hurt |
| Add singular-value truncation (ablation) | Convex formulation · CIFAR-10 | generalization performance | Helped |
| Add singular-value truncation (ablation) | Convex formulation · CIFAR-100 | generalization performance | Helped |
| Truncate too many singular values (k=10) (ablation) | Convex formulation · CNAE-9 | training and test performance: k=10 | Hurt |
| Retain all singular values (k=856; no truncation) (ablation) | Convex formulation · CNAE-9 | test performance: k=856 | Hurt |
| Use intermediate singular-value truncation (k=100 or k=200) (ablation) | Convex formulation · CNAE-9 | generalization performance: k=100 and k=200 | Helped |
| Vary minibatch size among the tested values (ablation) | Four-layer CNN with BN · CIFAR-100 binary classification, first two classes | training and test accuracy: 50, 100, 500, and 1000 | No clear effect |
| Use smaller batch sizes (ablation) | SGD and SGD-Truncated · CIFAR-10 and CIFAR-100 | generalization performance: 100, 500, 1000 | No clear effect |
| Train longer (ablation) | SGD, SGD-Truncated, Convex, and Convex-Truncated · CIFAR-10 and CIFAR-100 | test accuracy | Hurt |
| Change the optimizer from SGD to Adam (ablation) | BN and convex formulations · CIFAR-10 and CIFAR-100 | test accuracy | No clear effect |
Closest papers by meaning
- Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs (ICML 2021 · similarity 0.93)
- Optimization Theory for ReLU Neural Networks Trained with Normalization Layers (ICML 2020 · similarity 0.93)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.92)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.92)
- Norm matters: efficient and accurate normalization schemes in deep networks (NeurIPS 2018 · similarity 0.92)
- Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositions (ICML 2022 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.