Research map / Architectures and efficiency / RNNs and normalization
LCA: Loss Change Allocation for Neural Network Training
Janice Lan, Rosanne Liu, Hattie Zhou, Jason Yosinski
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Freeze the last layer at initialization (ablation) | CIFAR-ResNet with SGD · CIFAR-10 | train loss | Improved |
| Freeze the last layer at initialization (ablation) | CIFAR-ResNet with SGD · CIFAR-10 | test loss | Improved |
| Reduce the last layer learning rate by 10x (ablation) | CIFAR-ResNet with SGD · CIFAR-10 | layer LCA: 0.01 as opposed to 0.1 | Mixed |
| Freeze the first layer at initialization (ablation) | CIFAR-ResNet with SGD · CIFAR-10 | overall performance | No clear effect |
| Reduce the first layer learning rate by 10x (ablation) | CIFAR-ResNet with SGD · CIFAR-10 | layer LCA | No clear effect |
| Freeze the last layer at initialization (ablation) | CIFAR-AllCNN with SGD · CIFAR-10 | overall loss | Hurt |
| Use RK4 rather than first-order Taylor approximation (ablation) | LCA estimation · CIFAR-ResNet, SGD | total loss error: -1189.71\% (first-order) versus 0.0345\% (RK4) | Improved |
| Vary learning rate, momentum, or batch size within ranges where training remains effective (ablation) | CIFAR-ResNet with SGD · CIFAR-10 | helping parameters (%): 50.3\% to 51.6\% | No clear effect |
| Increase mini-batch size to 5000 (ablation) | CIFAR-ResNet with SGD · CIFAR-10 | test performance: 56.2\% | Hurt |
| Increase momentum to 0.99 (ablation) | CIFAR-ResNet with SGD · CIFAR-10 | train and test performance: 53.5\% | Hurt |
| Give the last layer a fresher-information advantage by reducing its momentum delay (ablation) | CIFAR-ResNet with SGD · CIFAR-10 | last-layer LCA | Improved |
Closest papers by meaning
- Visualizing the Loss Landscape of Neural Nets (NeurIPS 2018 · similarity 0.86)
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.86)
- Tilting the playing field: Dynamical loss functions for machine learning (ICML 2021 · similarity 0.85)
- A Loss Curvature Perspective on Training Instabilities of Deep Learning Models (ICLR 2022 · similarity 0.85)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.84)
- Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
