Research map / Architectures and efficiency / Pruning and efficient tuning
SeeDNorm: Self-Rescaled Dynamic Normalization
Wenrui Cai, Defa Zhu, Siyuan Qiao, Qingjie Liu, Qiyang Min
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace RMSNorm with SeeDNorm | OLMoE-1.3B · c4_en-validation | loss: 2.922 -> 2.900 | Improved |
| replace RMSNorm with SeeDNorm | OLMoE-7B · c4_en-validation | loss: 2.644 -> 2.631 | Improved |
| replace RMSNorm with SeeDNorm | OLMo2-550M · c4_en-validation | loss: 3.011 -> 3.008 | No clear effect |
| replace RMSNorm with SeeDNorm | OLMo2-1B · ARC-Easy | accuracy: 68.7 -> 70.0 | Improved |
| replace normalization with DyT | OLMoE-1.3B · c4_en-validation | loss: 2.922 -> 2.968 | Hurt |
| use SeeDNorm instead of LayerNorm | ViT-B · ImageNet-1K | Acc@1: 82.3 -> 82.7 | Improved |
| use unbounded activation (GeLU, Swish) for sigma (ablation) | SeeDNorm OLMoE-1.3B · training | convergence | Hurt |
| tanh vs other bounded activations (sigmoid, hardtanh) (ablation) | SeeDNorm OLMoE-1.3B · training loss | loss | Improved |
| initialize alpha to 10 instead of 1 (ablation) | SeeDNorm · c4_en-validation | loss: 2.900 -> 3.154 | Hurt |
| make alpha a vector rather than scalar (ablation) | SeeDNorm · c4_en-validation | loss: 2.909 -> 2.900 | Improved |
| use dot product instead of element-wise product of beta and x (ablation) | SeeDNorm · c4_en-validation | loss: 2.909 -> 2.900 | Improved |
| use multi-head SeeDNorm instead of single head (ablation) | SeeDNorm ViT-B · ImageNet-1K | Acc@1: fail to converge -> 82.7 | Improved |
Closest papers by meaning
- MetaNorm: Learning to Normalize Few-Shot Batches Across Domains (ICLR 2021 · similarity 0.89)
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.88)
- Impact of Layer Norm on Memorization and Generalization in Transformers (NeurIPS 2025 · similarity 0.86)
- Transformers without Normalization (CVPR 2025 · similarity 0.86)
- Learning in Compact Spaces with Approximately Normalized Transformer (NeurIPS 2025 · similarity 0.86)
- SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
