Research map / Architectures and efficiency / RNNs and normalization
Momentum Residual Neural Networks
Michael E. Sander, Pierre Ablin, Mathieu Blondel, Gabriel Peyré
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ResNet blocks with Momentum ResNet blocks using v₀ = 0 | ResNet-101 · CIFAR-10 | test accuracy: 95.15 ± 0.12 -> 95.1 ± 0.13 | No clear effect |
| Replace ResNet blocks with Momentum ResNet blocks using v₀ = 0 | ResNet-101 · CIFAR-100 | test accuracy: 76.86 ± 0.25 -> 76.39 ± 0.18 | Hurt |
| Use v₀ = f(x₀) instead of v₀ = 0 (ablation) | Momentum ResNet-101 · CIFAR-10 | test accuracy: 95.1 ± 0.13 -> 95.18 ± 0.06 | No clear effect |
| Use v₀ = f(x₀) instead of v₀ = 0 (ablation) | Momentum ResNet-101 · CIFAR-100 | test accuracy: 76.39 ± 0.18 -> 76.38 ± 0.42 | No clear effect |
| Vary the momentum parameter γ (ablation) | Momentum ResNet-101 · CIFAR-10 | test accuracy: never decreases by more than 3 % | No clear effect |
| Learn γ instead of fixing it (ablation) | Momentum ResNet-101 · CIFAR-10 | test accuracy | No clear effect |
| Transfer pretrained ResNet-152 parameters to Momentum ResNet-152 | ImageNet-pretrained ResNet-152 · ImageNet | top-1 error: 26.5 %; 23.5 % | Improved |
| Use Momentum ResNet-152 rather than ResNet-152 for fine-tuning | ImageNet-pretrained models · hymenoptera | fine-tuning convergence speed: twice as fast | Improved |
| Add momentum to first-order ODE blocks | CIFAR-10 ODE network · CIFAR-10 | test accuracy | Improved |
| Use Momentum ResNet dynamics instead of RevNet dynamics | Learned-ISTA · Lasso sparse coding test examples | test loss | Improved |
| Use Momentum ResNet instead of LISTA | Learned-ISTA · Lasso sparse coding test examples | test loss | Improved |
| Use Momentum ResNet rather than ResNet for separating nested rings | Depth-15 residual network · Four nested rings | class separability: 4 | Improved |
Closest papers by meaning
- MomentumRNN: Integrating Momentum into Recurrent Neural Networks (NeurIPS 2020 · similarity 0.89)
- Doubly infinite residual neural networks: a diffusion process approach (JMLR 2021 · similarity 0.88)
- Simplified Reversible Residual Networks (NeurIPS 2026 · similarity 0.88)
- Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows (ICML 2024 · similarity 0.86)
- Dr2Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning (CVPR 2024 · similarity 0.86)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
