Research map / Architectures and efficiency / RNNs and normalization
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
Haoming Meng, Anton Sugolov, Vardan Papyan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add window smoothing (alpha 0.1, Standard, Proj) to AdamW RLVR | GRPO on DeepSeek-R1-Distill-Qwen-1.5B · RLVR math reasoning average | pass@1 accuracy (%): 55.45 -> 57.60 | Improved |
| add window smoothing (alpha 0.1, Standard, Proj) to AdamW RLVR | GRPO on DeepSeek-R1-Distill-Qwen-1.5B · AIME24 | pass@1 accuracy (%): 35.00 -> 40.83 | Improved |
| add window smoothing (alpha 0.2, Standard, Proj) to AdamW RLVR | GRPO on DeepSeek-R1-Distill-Qwen-1.5B · AIME25 | pass@1 accuracy (%): 24.17 -> 30.00 | Improved |
| add window smoothing (alpha 0.2, Standard, Proj) to AdamW RLVR | GRPO on DeepSeek-R1-Distill-Qwen-1.5B · AMC23 | pass@1 accuracy (%): 78.12 -> 77.50 | Hurt |
| smooth only projection layers (Proj) instead of all block parameters (Full) at alpha 0.1 (ablation) | window smoothing in RLVR · RLVR math reasoning average | pass@1 accuracy (%): 56.80 -> 57.60 | Improved |
| increase smoothing strength from alpha 0.1 to 0.2 (Standard, Proj) (ablation) | window smoothing in RLVR · RLVR math reasoning average | pass@1 accuracy (%): 57.60 -> 57.38 | No clear effect |
| add window smoothing (alpha 0.2, Dir, Proj) to AdamW | ViT-B · CIFAR-100 | top-1 accuracy (%): 74.56 -> 75.62 | Improved |
| add window smoothing (alpha 0.1, Norm, Proj) to AdamW | ViT-B · CIFAR-100 | top-1 accuracy (%): 74.56 -> 75.44 | Improved |
| add window smoothing (alpha 0.2, Norm, Proj) to AdamW | U-ViT diffusion · CIFAR-10 FID@10k | FID: 6.58 -> 5.82 | Improved |
| add window smoothing (alpha 0.2, Norm, Proj) to AdamW | U-ViT diffusion · CIFAR-10 FID@50k | FID: 4.01 -> 3.74 | Improved |
| add window smoothing (alpha 0.1) to Adam+NorMuon pretraining | nanochat GPT, depth 24 and 30 · validation loss and CORE | validation loss, CORE | Improved |
| increase window smoothing strength alpha from 0.1 to 0.4 | ViT-B on CIFAR-100 · cosine similarity of layer contributions | mean cosine similarity | Improved |
Closest papers by meaning
- Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks (NeurIPS 2022 · similarity 0.88)
- Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity (ICLR 2020 · similarity 0.88)
- ASGO: Adaptive Structured Gradient Optimization (NeurIPS 2025 · similarity 0.87)
- On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.86)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
