Research map / Architectures and efficiency / RNNs and normalization
Scalable Optimization in the Modular Norm
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines the modular norm, the natural norm on the full weight space of any architecture built recursively from well-normed modules, and uses it to normalize updates of Adam/SGD so learning rates transfer across width and depth.
Problem
Learning rates that are optimal for small networks do not transfer as width and depth scale, and existing layer-wise correction factors for optimizers are optimizer-specific and hard to use; there is no principled norm on the full weight space of a neural network for normalizing updates and analyzin
Approach
Define a recursive "modular norm" over the module tree of any architecture (composition, concatenation, addition, scalar multiplication) with mass and sensitivity attributes, then normalize any base optimizer's updates in this norm so the learning rate transfers; also proves the loss gradient is Lipschitz in this norm with a recursively computed constant.
Key results
- Normed Adam and normed SGD allow learning rate transfer across width and depth for GPT on OpenWebText, and for ResMLP and ResNet on CIFAR-10.
- Normalization overhead measured at about 23% training time for a width-64 ResMLP (101 s vs 124 s over 10k steps on CIFAR-10).
- Optimal mass allocation for hidden layers transfers across depth, and the sharpness constant of a residual network is shown to be independent of depth for suitable block multipliers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add modular-norm update normalization | Adam and SGD · OpenWebText, GPT training across width and depth | learning-rate transfer | Helped |
| Add modular-norm update normalization | Adam and SGD · CIFAR-10, ResMLP and ResNet training across width and depth | learning-rate transfer | Helped |
| Add modular-norm update normalization | ResMLP width 64, 8 blocks, block-depth 2 · CIFAR-10, 10k steps | training time: 101 ± 1 seconds -> 124 ± 1 seconds | Hurt |
Limitations
- Well-normedness is only ensured at initialization and not enforced during training.
- Spectral-norm normalization via power iteration adds nontrivial overhead to training time.
- Some modules (ReLU, LayerNorm) are only approximately well-normed or require input conditions.
Experimental setup
Datasets: OpenWebText CIFAR-10
Benchmarks: OpenWebText CIFAR-10
Metrics: training loss test loss learning rate transfer across width and depth wall-clock time overhead
Baselines: Adam SGD nanoGPT with standard initialization Prodigy
Builds on: mu-P Tensor Programs Modular norm of deep relative trust (Xu et al.) RMS-RMS operator norm metrization (Yang et al.) PyTorch module tree
Closest papers by meaning
- Modular Duality in Deep Learning (ICML 2025 · similarity 0.90)
- Breaking Neural Network Scaling Laws with Modularity (ICLR 2025 · similarity 0.85)
- Generalization Bounds for Rank-sparse Neural Networks (NeurIPS 2025 · similarity 0.85)
- Unveiling The Matthew Effect Across Channels: Assessing Layer Width Sufficiency via Weight Norm Variance (NeurIPS 2024 · similarity 0.84)
- Learning in Compact Spaces with Approximately Normalized Transformer (NeurIPS 2025 · similarity 0.84)
- Smooth Min-Max Monotonic Networks (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.