AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

SimpleGPT: Improving GPT via A Simple Normalization Strategy

Marco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Weight normalization and decay interplay · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add SimpleNorm (normalize right after every linear layer) in place of QKNormLlama2 1B · C4 200K stepstraining loss: 2.478 -> 2.446Improved
add SimpleNorm instead of QKNormLlama2 7B · C4 20K stepstraining loss: +0.062 lower lossImproved
add SimpleNorm instead of QKNormLlama2 7B · C4 40K stepstraining loss: +0.077 lower lossImproved
add SimpleNorm instead of QKNormLlama2 7B · C4 60K stepstraining loss: +0.082 lower lossImproved
add SimpleNorm instead of QKNorm with 3x larger learning rateLlama3 8B · C4 20K stepstraining lossImproved
add SimpleNorm instead of QKNorm with 3x larger learning rateGPT-2 1.4B (nanoGPT) · validation loss, 100K stepsvalidation loss: +0.043 lower lossImproved
add QKNorm to GPT-2 1.4BGPT-2 1.4B (nanoGPT) · validation loss, 100K stepsvalidation lossNo clear effect
add SimpleNorm at learning rate 2e-3 and 2e-2 versus PreNorm+QKNorm (ablation)Llama2 1B · largest tolerable learning ratetraining stabilityImproved
add QKNorm to PreNorm at learning rate 2e-3 (ablation)Llama2 1B · largest tolerable learning ratetraining stabilityImproved
add QKNorm at low learning rate 2e-4 (ablation)Llama2 1B · C4 1Btraining lossNo clear effect
add SimpleNorm across learning rates 2e-4, 2e-3, 2e-2 (ablation)Llama2 1B · C4 1Btraining lossImproved
add SimpleNorm instead of QKNormLlama3 8B · training time per steptime per step (ms): 1553 -> 1603Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy