Research map / Learning theory and algorithms / Kernels and regression theory
Truth or backpropaganda? An empirical investigation of deep learning theory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Bias parameter norm toward a non-zero value instead of shrinking it toward zero (ablation) | ResNet-18, DenseNet-40, MobileNetV2, ResNet-20 Fixup, and 4-layer MLP · CIFAR-10, normalized and non-normalized | accuracy: 94.06 -> 94.86; 92.27 -> 92.49; 92.88 -> 93.50; 94.76 -> 94.99; 91.42 -> 91.55; 58.95 -> 59.13 | Helped |
| Bias parameter norm toward a non-zero value instead of shrinking it toward zero (ablation) | ResNet-18, DenseNet-40, and ResNet-20 Fixup · CIFAR-100, normalized | accuracy: 74.66 -> 75.90; 68.98 -> 69.24; 65.08 -> 65.58 | Helped |
| Promote high-rank operators instead of using standard adversarial training (ablation) | ResNet-18 · CIFAR-10, adversarial training | robust accuracy (ε = 8/255): 35.38 -> 35.55 | Helped |
| Promote low-rank operators instead of using standard adversarial training (ablation) | ResNet-18 · CIFAR-10, adversarial training | robust accuracy (ε = 8/255): 35.38 -> 33.68 | Hurt |
| Promote high rank or low rank during fine-tuning (ablation) | ResNet-18 · CIFAR-10, natural training | clean accuracy: 94.66 -> 93.66; 94.66 -> 94.44 | No clear effect |
| Initialize from a trained linear-classifier solution (ablation) | MLP · CIFAR-10 | training loss: 0.0061 -> 1.5699 | Hurt |
| Initialize biases to +20 (ablation) | MLP · CIFAR-10 | training loss: 0.0061 -> 2.3301 | Hurt |
| Initialize biases from a uniform distribution over (-50,50) (ablation) | MLP · CIFAR-10 | training loss: 0.0061 -> 2.3153 | Hurt |
| Remove batch normalization or skip connections from WideResNet (ablation) | WideResNet · CIFAR-10, NTK correlation before versus after training | NTK correlation | Mixed |
Closest papers by meaning
- The Loss Surface of Deep and Wide Neural Networks (ICML 2017 · similarity 0.90)
- Small nonlinearities in activation functions create bad local minima in neural networks (ICLR 2019 · similarity 0.89)
- A Convergence Theory for Deep Learning via Over-Parameterization (ICML 2019 · similarity 0.89)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.88)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.88)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.