Research map / Optimization / Implicit bias and generalization
Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that minibatch SGD noise causes chaotic rollout-reward oscillations in behavior cloning (gradient variance amplification, GVA), and that exponential moving average of iterates mitigates it in continuous control and autoregressive language generation.
Problem
Behavior cloning with deep networks shows large oscillations in long-horizon rollout reward across nearby checkpoints even when the behavior cloning validation loss decreases smoothly, making checkpoint selection unreliable.
Approach
Attributes the oscillations to gradient variance amplification: minibatch SGD noise is benign for the one-step BC loss but is amplified through closed-loop dynamics; an exponential moving average of iterates suppresses it without changing the learning rate schedule or batch size.
Key results
- EMA of iterates strongly reduces rollout-reward oscillations across architectures and environments, without hurting performance in any tested setup (R4).
- Shallower/variance-reduction interventions (aggressive LR decay, larger batch) reduce oscillations but at large compute cost; regularization and architecture changes do not remove them.
- In 270M-parameter TinyStories language models, EMA stabilizes training, enables training without LR decay, and yields the lowest-perplexity model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EMA iterate averaging | Behavior cloning policies · MuJoCo continuous control, including Walker2d-v4 | rollout reward oscillations | Helped |
| Aggressively decay the learning rate (ablation) | Behavior cloning training · Walker2d-v4 | rollout reward oscillations | Helped |
| Increase batch size through gradient accumulation (ablation) | Behavior cloning training · Walker2d-v4 | rollout reward oscillations | Helped |
| Change model architecture or scale (ablation) | Behavior cloning policies · Walker2d-v4 | rollout reward oscillations | No clear effect |
| Increase standard regularization (ablation) | Behavior cloning policies · Walker2d-v4 | rollout reward oscillations | No clear effect |
| Use a larger momentum coefficient (ablation) | Behavior cloning optimizer · MuJoCo continuous control | training stability | Helped |
| Add EMA iterate averaging without changing the learning-rate schedule or batch size | Behavior cloning policies · MuJoCo continuous control | rollout reward oscillations | Helped |
| Add EMA iterate averaging | 270M-parameter, 12-layer Transformer · TinyStories | generalization | Helped |
| Train with EMA rather than relying on full learning-rate decay | Autoregressive language model · TinyStories | perplexity | Helped |
Surprising findings
- Rollout oscillations persist even with fresh i.i.d. trajectories at each step (infinite data), indicating an algorithmic rather than statistical cause.
- Common regularizers and architecture changes do not mitigate GVA; only variance reduction (LR decay, large batch, EMA) helps.
Limitations
- Theoretical results are for toy one-dimensional and linear settings; the theory is only a working model for GVA.
- Language experiments are preliminary and use a small synthetic corpus (TinyStories), with rollout oscillations measured via disagreement in long-horizon rollouts rather than a canonical reward.
- Experiments focus on fully offline deterministic MuJoCo settings with expert-generated data.
Experimental setup
Datasets: TinyStories Wikipedia MuJoCo expert trajectories (Walker2d-v4, Hopper-v4, HalfCheetah-v4, Humanoid-v4, Ant-v4)
Benchmarks: Walker2d-v4 Hopper-v4 HalfCheetah-v4 Humanoid-v4 Ant-v4 TinyStories Wikipedia
Metrics: rollout reward validation behavior cloning loss validation loss / perplexity
Baselines: vanilla SGD/AdamW training without EMA learning rate decay larger batch size via gradient accumulation
Builds on: Soft Actor Critic exponential moving average DART DAgger
Closest papers by meaning
- Strength of Minibatch Noise in SGD (ICLR 2022 · similarity 0.89)
- Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks (NeurIPS 2023 · similarity 0.88)
- Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning (ICML 2023 · similarity 0.88)
- Chaotic Dynamics are Intrinsic to Neural Network Training with SGD (NeurIPS 2022 · similarity 0.88)
- Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning (ICML 2024 · similarity 0.87)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.