Research map / Architectures and efficiency / RNNs and normalization
SimpleGPT: Improving GPT via A Simple Normalization Strategy
Marco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SimpleNorm (normalize right after every linear layer) in place of QKNorm | Llama2 1B · C4 200K steps | training loss: 2.478 -> 2.446 | Improved |
| add SimpleNorm instead of QKNorm | Llama2 7B · C4 20K steps | training loss: +0.062 lower loss | Improved |
| add SimpleNorm instead of QKNorm | Llama2 7B · C4 40K steps | training loss: +0.077 lower loss | Improved |
| add SimpleNorm instead of QKNorm | Llama2 7B · C4 60K steps | training loss: +0.082 lower loss | Improved |
| add SimpleNorm instead of QKNorm with 3x larger learning rate | Llama3 8B · C4 20K steps | training loss | Improved |
| add SimpleNorm instead of QKNorm with 3x larger learning rate | GPT-2 1.4B (nanoGPT) · validation loss, 100K steps | validation loss: +0.043 lower loss | Improved |
| add QKNorm to GPT-2 1.4B | GPT-2 1.4B (nanoGPT) · validation loss, 100K steps | validation loss | No clear effect |
| add SimpleNorm at learning rate 2e-3 and 2e-2 versus PreNorm+QKNorm (ablation) | Llama2 1B · largest tolerable learning rate | training stability | Improved |
| add QKNorm to PreNorm at learning rate 2e-3 (ablation) | Llama2 1B · largest tolerable learning rate | training stability | Improved |
| add QKNorm at low learning rate 2e-4 (ablation) | Llama2 1B · C4 1B | training loss | No clear effect |
| add SimpleNorm across learning rates 2e-4, 2e-3, 2e-2 (ablation) | Llama2 1B · C4 1B | training loss | Improved |
| add SimpleNorm instead of QKNorm | Llama3 8B · training time per step | time per step (ms): 1553 -> 1603 | Hurt |
Closest papers by meaning
- Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency (NeurIPS 2025 · similarity 0.88)
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.88)
- Learning in Compact Spaces with Approximately Normalized Transformer (NeurIPS 2025 · similarity 0.87)
- CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs (NeurIPS 2026 · similarity 0.87)
- Learning Rate Transfer for Hybrid Transformer-SSM Architectures (NeurIPS 2026 · similarity 0.87)
- Taming Transformer Without Using Learning Rate Warmup (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
