Research map / Large language models / Language, safety and interpretability
LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper shows that loss-to-loss scaling laws between pretraining/validation and downstream test losses follow shifted power laws, and that pretraining data is the dominant factor shaping them while tokenizer, architecture (Llama vs Mamba), model size, context length, and optimizer have minor effects.
Problem
Loss-to-loss scaling laws relate losses across pretraining/validation datasets and downstream tasks, but it is unknown which training design choices causally shape their form, which matters for predicting generalization and tuning downstream performance.
Approach
Runs controlled interventions over pretraining data, tokenizer, architecture, model size, context length, and optimizer settings (over 6000 checkpoints) and fits shifted power laws between average validation and test losses, finding that only pretraining data substantially shifts these curves while the other factors have little effect.
Key results
- Loss-to-loss relationships across architectures and training settings follow shifted power laws (L_y = K(L_x - E_x|p)^kappa + E_y|p).
- Area-between-curves analysis shows pretraining data has the largest effect (0.22 vs base) while tokenizer, architecture, optimizer, size, and context each have 0.00-0.03 effects (architecture 0.02 vs base).
- Point-wise comparison: Mamba 421M and Llama 417M trained on FineWeb-Edu show nearly identical average validation loss (3.63 vs 3.58) and average test loss (3.88 vs 4.04).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Change pretraining data while matching architecture and tokenizer | LLM loss-to-loss scaling · Validation-to-test and train-to-test loss scaling | loss-to-loss scaling curve: 0.22 | Mixed |
| Change tokenizer while matching pretraining data and architecture | LLM loss-to-loss scaling · BPB curves | loss-to-loss scaling curve: 0.01 | No clear effect |
| Change architecture while matching pretraining data and tokenizer | LLM loss-to-loss scaling · Train-to-train and train-to-test | loss-to-loss scaling curve: 0.02 | No clear effect |
| Vary model size | Llama and Mamba · Train-to-train loss scaling | loss-to-loss scaling curve: 0.02 | No clear effect |
| Vary context length among 1024, 2048, and 3076 tokens | LLM loss-to-loss scaling · Train-to-train loss scaling | loss-to-loss scaling curve: 0.02 | No clear effect |
| Vary optimizer, schedule, learning rate, or weight decay | LLM loss-to-loss scaling · Train-to-train loss scaling | loss-to-loss scaling coefficients: 0.02 | No clear effect |
| Use Adam optimizer | Models in the optimization-settings experiments · Training | convergence | Hurt |
Surprising findings
- Architecture differences between transformer (Llama) and state-space model (Mamba) have little effect on loss-to-loss scaling when trained on the same data.
- Adam-optimized models generally did not converge in their training setup and were excluded from analysis.
Limitations
- Limited compute prevented training many models from scratch; intermediate checkpoints used as proxies for smaller-token models, which may affect compute-to-loss fits.
- Only tested up to ~420M parameter models and context lengths up to 3072; larger scale effects not verified.
- Weight tying discrepancy with Hugging Face ablation models was not resolved.
Experimental setup
Datasets: FineWeb-Edu C4 The Pile UC The Pile The Pile Deduped RefinedWeb SlimPajama HellaSwag COPA WinoGrande PIQA SocialIQA
Benchmarks: HellaSwag COPA WinoGrande PIQA SocialIQA CommonsenseQA MMLU ARC-Easy ARC-Challenge
Metrics: cross-entropy loss (cloze formulation) bits-per-byte fitted shifted power law parameters area between fitted curves
Builds on: Chinchilla scaling laws Accuracy on the line Brandfonbrener et al. loss-to-loss scaling
Closest papers by meaning
- Position: Enough of Scaling LLMs! Lets Focus on Downscaling (ICML 2025 · similarity 0.93)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.92)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.92)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.91)
- Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies (NeurIPS 2024 · similarity 0.91)
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.