Research map / Architectures and efficiency / RNNs and normalization
Accelerating Training with Neuron Interaction and Nowcasting Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NiNo, a graph-neural-network meta-model that periodically nowcasts future neural network parameters on neural graphs of Transformers, speeding up Adam/AdamW training by ~49% on average across nine vision and language tasks.
Problem
Accelerating neural network training by periodically predicting future parameters, where prior weight nowcaster networks (WNNs) ignore neuron connectivity and mis-model neuron permutation symmetry in Transformers.
Approach
Represents parameter trajectories as neural graphs with a corrected multi-head self-attention construction (head-restricted permutations, separate q/k/v edge features), processes them with a GNN that predicts multi-step future edge features (direct multi-step forecasting), and uses layerwise scaling and decaying horizon k at inference.
Key results
- NiNo achieves an average 48.9% reduction in steps to reach Adam's target performance across nine tasks, versus 39.8% for WNN+ and 26.5% for WNN.
- On a Llama3-style 111M Transformer, AdamW+NiNo reaches AdamW's 14k-step validation perplexity (24.4) in about 30% fewer steps.
- On ImageNet with a small ViT, AdamW+NiNo improves validation top-1 accuracy from 49.6% to 50.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NiNo instead of the base optimizer alone | Adam/AdamW training · Nine vision and language tasks | steps to target performance: 48.9% average speedup | Helped |
| use NiNo instead of WNN+ | training across the nine tasks · Nine vision and language tasks | average speedup (%): 39.8 -> 48.9 | Helped |
| replace the naive Transformer graph with the symmetry-aware neural graph (ablation) | NiNo · LM1B/2-32 | speedup (%): 12.8 -> 48.9 | Helped |
| add layerwise parameter scaling (ablation) | NiNo · Average across tasks | speedup (%): 32.1 -> 48.9 | Helped |
| add k-decay with p=2 (ablation) | NiNo · Average across tasks | speedup (%): 40.8 -> 48.9 | Helped |
| add LPE node features (ablation) | NiNo · Average across tasks | speedup (%): 47.5 -> 48.9 | Helped |
| add word positional encoding (ablation) | NiNo · Average across tasks | speedup (%): 45.4 -> 48.9 | Helped |
| add edge-type embeddings (ablation) | NiNo · Average across tasks | speedup (%): 45.7 -> 48.9 | Helped |
| use context length c=3 instead of c=5 (ablation) | NiNo · Average across tasks | speedup (%): 48.9 -> 29.8 | Hurt |
| use context length c=7 instead of c=5 (ablation) | NiNo · Average across tasks | speedup (%): 48.9 -> 44.0 | Hurt |
| use context length c=10 instead of c=5 (ablation) | NiNo · Average across tasks | speedup (%): 48.9 -> 34.0 | Hurt |
| reduce GNN width to D=32 (ablation) | NiNo · Average across tasks | speedup (%): 48.9 -> 42.6 | Hurt |
Surprising findings
- Simple Linefit baselines outperform the learned WNN on several tasks, suggesting WNN learned a very simple rule.
- NiNo's PCA trajectory closely resembles a 2x faster AdamW despite using Adam as base optimizer, hinting at implicit regularization.
Limitations
- Meta-model is underfitting when trained on all checkpoints, and scaling beyond depth 3 or width 128 was harder to train.
- Speedups on the largest and Llama-style models are smaller (around 15% for 29M and 14-36% depending on learning rate), and the ImageNet gain is small.
- Neural graph construction for Transformers required custom design and the naive graph underperforms on out-of-distribution language tasks.
Experimental setup
Datasets: FashionMNIST CIFAR-10 CIFAR-100 LM1B WikiText103 ImageNet
Benchmarks: FashionMNIST CIFAR-10 CIFAR-100 LM1B WikiText103 ImageNet
Metrics: relative reduction in steps to reach target validation accur validation perplexity top-1 accuracy wall-clock overhead
Baselines: Adam AdamW Linefit Linefit+ WNN WNN+ L2O/FM16 NiNo (naive graph)
Builds on: Weight nowcaster networks (WNN) Neural graphs (Kofinas et al.) Learning to optimize (L2O) Adam Linefit
Closest papers by meaning
- Learning to Boost Training by Periodic Nowcasting Near Future Weights (ICML 2023 · similarity 0.89)
- How Do Adam and Training Strategies Help BNNs Optimization (ICML 2021 · similarity 0.85)
- Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam (ICLR 2023 · similarity 0.84)
- Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (ICLR 2022 · similarity 0.84)
- NDOT: Neuronal Dynamics-based Online Training for Spiking Neural Networks (ICML 2024 · similarity 0.83)
- Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.