Research map / Architectures and efficiency / RNNs and normalization

Progressive Residual Warmup for Language Model Pretraining

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ProRes, a training-time residual warmup that scales each layer's residual branch by a scalar rising from 0 to 1 with deeper layers warming up later, improving pretraining perplexity and downstream accuracy across normalization schemes and depths.

Problem

Transformer pretraining is unstable and slow to converge, and deep layers may contribute before upstream representations stabilize; existing stabilization methods are mostly applied at initialization and are not training-phase-aware.

Approach

Multiply each layer's residual branch by a scalar alpha(l,t) that warms up linearly from 0 to 1, with warmup length increasing with layer index, so shallow layers learn first and deeper layers engage later; applied on top of Pre-LN, Post-LN, Sandwich-LN, DeepNorm and LayerNorm Scaling.

Key results

• On 1.3B models, ProRes improves Pre-LN zero-shot average accuracy from 43.42 to 45.00 across reasoning benchmarks, and average C4 perplexity drops by roughly 0.4 across architectures.
• Post-LN with ProRes at 1.3B drops C4 perplexity from 11.62 to 10.53 and LAMBADA perplexity from 33.53 to 22.91.
• ProRes (linear schedule) keeps Pre-LN depth scaling best up to 120 layers while maintaining near-zero loss spikes, per the paper's spike-score analysis.

Surprising findings

• Equal (simultaneous) schedules and reverse (deep-first) schedules underperform or diverge, showing that activation order across depth matters more than warmup alone.
• LayerNorm Scaling gains from ProRes shrink at larger scale, with the authors hypothesizing that the extra downscaling of deep residuals suppresses learning.

Limitations

• Schedule and warmup length T were not tuned for the main experiments and the best schedule is architecture-dependent.
• Gains on some architectures at larger scale are small (e.g., LNS at 1.3B shows about 0.04 perplexity change), and the 7B experiments are limited to Pre-LN.

Experimental setup

Datasets: C4-en ClimbMix WikiText LAMBADA

Benchmarks: C4-en test perplexity PIQA SIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenBookQA RACE LAMBADA MMLU WikiText perplexity

Metrics: perplexity zero-shot accuracy loss spike score

Baselines: Pre-LN Pre-LN (DS-Init) Pre-LN (Scaled Init) Sandwich-LN LayerNorm Scaling Post-LN DeepNorm

Builds on: Pre-LN Post-LN DeepNorm LayerNorm Scaling WSD learning rate scheduler

Closest papers by meaning

• GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.89)
• Preparing Lessons for Progressive Training on Language Models (AAAI 2024 · similarity 0.88)
• Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training (NeurIPS 2025 · similarity 0.88)
• The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.88)
• Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.88)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.88)