Research map / Architectures and efficiency / RNNs and normalization

GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling

NeurIPS 2025 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GPAS, which scales down forward activations via a learnable SiLU gate while stop-gradient keeps backward gradients unchanged, reducing activation variance growth in Pre-LN LLMs and improving pretraining convergence and downstream SFT accuracy across 71M–1B models.

Problem

Pre-LN Transformers suffer exponential growth of activation variance across depth, so the shortcut dominates sub-layer outputs and deeper layers are underutilized, slowing convergence and reducing parameter efficiency.

Approach

Insert a learnable scalar gate after each sub-layer that scales the residual stream by (1 - SiLU(alpha)) in the forward pass, while a stop-gradient operator makes the backward Jacobian identity so gradients are not downscaled; the same plug-in is applied to DeepNorm, Sandwich-LN, Mix-LN and LayerNorm Scaling.

Key results

• Pre+GPAS reduces 350M perplexity from 21.35 to 20.35 and 1B perplexity from 16.92 to 16.25 relative to Pre-LN.
• LNS+GPAS achieves the best 1B pretraining perplexity of 14.88 versus 15.65 for LNS alone.
• On 1B SFT with LISA, LNS+GPAS reaches 45.83% average accuracy versus 44.65% for LNS; DeepNorm+GPAS avoids the divergence seen in vanilla DeepNorm.

Measured effects reported in the paper

Surprising findings

• Identity or Tanh activation for the gate gives slightly better 350M perplexity than SiLU, but the advantage does not hold at 1B.
• Without the stop-gradient operator, GPAS gives essentially no perplexity gain over Pre-LN (21.34 vs 21.35), even though it still reduces activation variance by about 50%.

Limitations

• Experiments on pretraining are limited to models up to 1B (with a single 7B run truncated to 40K steps), constrained by compute.
• Learnable gates can fluctuate aggressively (e.g., gradient spike around step 10K), requiring gradient clipping on gates in some runs.
• GPAS is intended for training from scratch and is likely suboptimal when applied to models pretrained without it.

Experimental setup

Datasets: C4 Commonsense170K

Benchmarks: MMLU WinoGrande PIQA OpenBookQA HellaSwag BoolQ ARC-easy

Metrics: perplexity accuracy

Baselines: Post-LN DeepNorm Pre-LN Sandwich-LN Mix-LN LayerNorm Scaling

Builds on: Pre-LN Post-LN DeepNorm Sandwich-LN LayerNorm Scaling

Closest papers by meaning

• The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.90)
• Progressive Residual Warmup for Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.89)
• La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.89)
• Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
• Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN (ICLR 2025 · similarity 0.89)