Research map / Architectures and efficiency / RNNs and normalization

Modular Duality in Deep Learning

ICML 2025 · Architectures and efficiency › RNNs and normalization · approach: Theory of depth and width · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes modular dualization, a recursive construction of duality maps for general neural networks from layerwise operator norms, with GPU-friendly rectangular Newton-Schulz algorithms for Linear, Embed, and Conv2D layers.

Problem

Gradient descent subtracts a dual vector from primal weights without a duality map; the paper seeks a principled, scalable way to construct duality maps (steepest descent under norms) for general neural architectures with heterogeneous curvature.

Approach

Assign operator norms to layers based on input/output semantics, then recursively combine layerwise norms via the modular norm (weighted max over sub-modules) to induce a full-network duality map; derive atomic duality maps for Linear (RMS-to-RMS), Embed (l1-to-RMS) and Conv2D layers, computed via rectangular Newton-Schulz iteration.

Key results

• Shows that the Linear duality map recovers spectral update scaling underlying muP and the Shampoo-without-accumulation mapping G -> UV^T.
• Derives closed-form duality maps for Linear (sqrt(d_out/d_in) UV^T), Embed (column RMS normalization) and Conv2D (per-kernel-slice scaled UV^T) layers.
• Reports that a variant of the methods was used to set NanoGPT training speed records (no quantitative results given in this text).

Surprising findings

• Duality maps ramp up the stable rank of updates, so weights may move a non-trivial relative amount at large width even in Frobenius norm, provided batch size is not too small.

Limitations

• No empirical experiments reported in this text; claims about NanoGPT speedrun records are not quantified here.
• Sketching-based approximations may fail to capture small singular values, and inverse-root iterations are unstable for ill-conditioned matrices.

Experimental setup

Baselines: Shampoo maximal update parameterization spectral descent duality structure gradient descent

Builds on: modular norm spectral descent duality structure gradient descent Shampoo maximal update parameterization

Closest papers by meaning

• Scalable Optimization in the Modular Norm (NeurIPS 2024 · similarity 0.90)
• Breaking Neural Network Scaling Laws with Modularity (ICLR 2025 · similarity 0.84)
• Tensorion: A Tensor-Aware Generalization of the Muon Optimizer (NeurIPS 2026 · similarity 0.84)
• On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.84)
• Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data (NeurIPS 2025 · similarity 0.84)
• Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.84)