Research map / Architectures and efficiency / RNNs and normalization
Learning to Boost Training by Periodic Nowcasting Near Future Weights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add periodic short-term nowcasting of weights (predict 5 epochs ahead) with WNN | Vanilla CNN training · CIFAR10, time to 59% val acc | speed up: ×2.09 vs naive | Helped |
| Replace linear curve fitting with a learned WNN predictor | Nowcasting of future weights · CIFAR10 Vanilla CNN, time to 59% val acc | speed up: ×2.09 vs ×1.33 for CF (Linear) | Helped |
| Pre-trained WNN (trained only on image classification) applied to unseen tasks without fine-tuning | Unseen tasks · Table 5 average over 7 tasks | speed up: ×1.25 avg vs Naive ×1.00 | Helped |
| Linear curve fitting and Introspection applied to unseen tasks | Unseen tasks · Table 5 average over tasks | speed up: ×0.99 (CF Linear), ×0.94 (Introspection) avg | Hurt |
| Replace L2O-Amalgam / L2O / HyperNetworks as acceleration method | Vanilla CNN · CIFAR10, time to 59% val acc | speed up: L2O-Amalgam ×0.79; HyperNetworks ×1.04; L2O ×1.20 | Hurt |
| Remove WNN components (periodic nowcasting, normalized residual, two-stream, activation network) all together (ablation) | WNN · CIFAR10 Vanilla CNN, time to 59% val acc | converge time (sec): 44.43 sec without components vs 25.27 sec with full WNN | Helped |
| Add periodic short-term forecasting (vs. none) (ablation) | WNN · CIFAR10 Vanilla CNN, epochs to 59% | converge epochs: 37 -> 34 epochs | Helped |
| Add normalized residual, two-stream design and activation network stepwise on top of periodic forecasting (ablation) | WNN · CIFAR10 Vanilla CNN, epochs to 59% | converge epochs: 34 -> 31 -> 29 -> 21 epochs | Helped |
| Use input length 5 and skip length 5 instead of other settings (ablation) | WNN · CIFAR10 Vanilla CNN, time to 59% | converge time (sec): 25.27 sec vs 31.35 sec for input 3 / skip 3 (best other tested) | Helped |
| Increase skip length beyond 5 epochs (e.g., 7 or 10) (ablation) | WNN · CIFAR10 Vanilla CNN, time to 59% | converge time (sec): 30.07 (skip 7), 31.29 (skip 10) vs 25.27 (skip 5) at input 5 | Hurt |
| Apply WNN to MobileNetV2 ImageNet, DeepLabV3+, OpenPose, BERT, DDPG, DDIM etc. (unseen tasks) | WNN · Table 5 per-task speed up | speed up: ranges ×1.15 (LM) to ×1.43 (pose), with Introspection/CF below 1 on some tasks | Mixed |
| Apply WNN to unseen optimizers and Fashion MNIST | WNN · Fashion MNIST ResNet32 (Appendix E) | convergence vs naive: qualitative only | Helped |
Closest papers by meaning
- Accelerating Training with Neuron Interaction and Nowcasting Networks (ICLR 2025 · similarity 0.89)
- Initializing Models with Larger Ones (ICLR 2024 · similarity 0.86)
- RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks (ICLR 2025 · similarity 0.84)
- Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations (NeurIPS 2024 · similarity 0.84)
- Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.84)
- Pretraining with Random Noise for Fast and Robust Learning without Weight Transport (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.