Research map / Architectures and efficiency / RNNs and normalization

Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies data-constrained multi-epoch LM pretraining, proposing MIR (an auxiliary next-token loss on randomly masked inputs added to strongly weight-decayed AR training) and the SoftQ coupled scaling law, which fits repeated-data loss better than Chinchilla, Quanta, and Muennighoff-style laws.

Problem

Pretraining is becoming data-constrained and compute-rich, so models train for many epochs over a fixed unique corpus; it is unclear how to regularize AR models beyond weight decay and whether additive Chinchilla-style scaling laws can describe the loss surface in this regime.

Approach

Adds an auxiliary next-token loss on randomly masked inputs (dLLM-style masking) to strongly regularized AR pretraining without architectural changes, and proposes SoftQ, a five-parameter scaling law that couples model and data size via a soft bottleneck with exponent rho, nesting Quanta when rho=1.

Key results

• MIR lowers 1.4B validation loss on DCLM 100M from 3.347 to 3.317 (mean over seeds), with gains growing from about 0.006 at 72M to about 0.03 at the largest models.
• At 1.4B, MIR gains +10.18 points on BoolQ and +2.20 on SciQ over the strongly regularized baseline, winning six of eight zero-shot metrics.
• SoftQ achieves held-out 400M RMSE of 0.00595 versus 0.03106 for Chinchilla, and MIR is estimated at about 1.28-1.34x data efficiency at 200M-400M unique tokens.

Surprising findings

• Applying the AR-tuned large weight decay to dLLMs substantially lowers their loss, and once both are strongly regularized their losses become comparable, suggesting the previously reported AR-dLLM gap is largely due to insufficient AR regularization.
• The Chinchilla-predicted loss gap between data budgets is constant across model sizes, but the observed gap grows with model size.

Limitations

• Experiments are limited to up to 1.4B parameters and 400M unique tokens, far below frontier scale.
• Relies on heavy per-cell hyperparameter search, and architecture and optimizer were held fixed.
• MIR doubles forward-pass compute per step; the paper frames it as data efficiency rather than compute efficiency.

Experimental setup

Datasets: DCLM-Pool Stack-V2

Benchmarks: DCLM validation loss Stack-V2 validation loss ARC-Easy BoolQ HellaSwag PiQA RACE SciQ Lambada

Metrics: validation loss RMSE MAE AIC accuracy perplexity data efficiency ratio

Baselines: Strongly regularized AR Multi-epoch AR Chinchilla Quanta Muennighoff-style law masked diffusion LM

Builds on: Chinchilla scaling law Quanta scaling law Muennighoff data-constrained scaling laws masked diffusion language models WSD learning-rate schedule

Closest papers by meaning

• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.91)
• An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.90)
• Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.90)
• Scaling Laws for Precision (ICLR 2025 · similarity 0.90)
• Bridging Compute- and Data-Optimal Pretraining (NeurIPS 2026 · similarity 0.90)
• Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.89)