Research map / Optimization / Implicit bias and generalization
Transformative or Conservative? Conservation laws for ResNets and Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives and characterizes conservation laws for gradient flows on building blocks of ResNets and Transformers (shallow ReLU/linear and convolutional nets, attention, softmax classifier, residual blocks) and shows they approximately persist under SGD with O(step-size^2) error.
Problem
Conservation laws of gradient-flow training dynamics are well understood only for shallow ReLU and linear networks; their structure for modern architectures such as convolutional ResNets and Transformers, and whether they survive discrete SGD, is unknown.
Approach
Uses a Lie-algebra/orthogonality characterization of conservation laws to show that shallow building blocks (multi-channel convolution, single-head attention, softmax classifier) have exactly the known or newly identified conservation laws, that residual blocks share their block's laws, and that block-local laws in deep ResNets/Transformers reduce to the isolated block analysis.
Key results
- Shallow multi-channel convolutional ReLU networks have exactly c_1 independent conservation laws, sum_k ||u_{k,j}||^2 - sum_i ||v_{j,i}||^2, and no others.
- For a single attention layer the conservation laws are functions of QQ^T - KK^T and VV^T - OO^T, and residual blocks have the same laws as the block without skip connection.
- Numerically, the conservation error of ResNet-18 on CIFAR-10 and of a transformer on IMDb scales as O(step-size^2) under SGD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use discrete SGD with learning rates from $10^{-3}$ to $5 \times 10^{-3}$ | ResNet-18 · CIFAR-10 | relative conservation error: O(\text{step-size}^2) | Mixed |
| Train with SGD | Transformer · IMDb sentiment analysis | conservation error: O(\text{step-size}^2) | Mixed |
| Apply masking versus no masking | Transformer · IMDb sentiment analysis | conservation error | No clear effect |
Surprising findings
- For ResNets with m=1 the numerics suggest additional conservation laws beyond the block-local ones, confirmed by a two-block example on a restricted parameter domain.
- No additional conservation laws appear numerically for ResNets with m>1 and q=2 blocks.
Limitations
- Transformer normalization layers are not analyzed.
- Multi-head attention completeness of conservation laws is left open; only a partial set is given.
- SGD error bound relies on a bounded-gradient-in-expectation assumption that may not hold for deep networks.
Experimental setup
Datasets: CIFAR-10 IMDb
Metrics: conservation error |(h(theta_k)-h(theta_0))/h(theta_0)| dimension of span of gradients
Builds on: balancedness conservation laws for ReLU and linear networks Lie algebra characterization of conservation laws (Marcotte et al.) Frobenius theorem
Closest papers by meaning
- Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows (ICML 2024 · similarity 0.92)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.89)
- Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (ICLR 2025 · similarity 0.89)
- On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning (NeurIPS 2025 · similarity 0.88)
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.87)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.