Research map / Large language models / Language, safety and interpretability
Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes fully online, loss-based instance-level reweighting (LinUpper, Quadratic, Extremes) for LLM pretraining, with a convergence analysis showing down-weighting low-loss samples tightens bounds, evaluated on GPT2 models and 1.4B/7B Llama.
Problem
Standard LLM pretraining treats all samples equally, and existing reweighting methods are group-level, static, or require stored per-sample statistics or validation-set bilevel optimization, which are impractical for single-epoch web-scale pretraining.
Approach
Reweight each minibatch sample online by a bounded, analytic transform of its normalized current loss (with a temperature r annealed during training and a cap of 2/b on weights), so no extra forward/backward passes or stored statistics are needed; a convergence theorem justifies down-weighting low-loss samples.
Key results
- On GPT2-medium with uniform domain sampling, LinUpper gives 5-shot mean accuracy 40.9 vs 39.4 for uniform across LogiQA, LogiQA 2, SciQ and PiQA.
- With 1.4B Llama, LinUpper improves average over 19 tasks from 46.56 to 47.55 (LUR +0.66, QA +1.72).
- With 7B Llama, LinUpper improves average across 19 tasks from 49.79 to 51.14 (LUR +1.44, QA +1.16).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LinUpper loss-based instance reweighting over uniform sampling | GPT2-medium pretraining · SlimPajama-6B 7 domains, hold-out perplexity | mean perplexity: 3.13 -> 3.10 | Helped |
| Add LinUpper loss-based instance reweighting over uniform sampling | GPT2-mini and GPT2-small pretraining · SlimPajama-6B 7 domains, hold-out perplexity | mean perplexity: 3.32 -> 3.30 (mini); 3.15 -> 3.13 (small) | Helped |
| Use Quadratic reweighting instead of uniform (ablation) | GPT2 pretraining · SlimPajama-6B hold-out perplexity | mean perplexity: 3.13 -> 3.19 (medium); 3.15 -> 3.23 (small) | Hurt |
| Use Extremes reweighting instead of uniform (ablation) | GPT2 pretraining · SlimPajama-6B hold-out perplexity | mean perplexity: 3.13 -> 3.21 (medium) | Hurt |
| Add LinUpper on top of uniform 5-shot sampling | GPT2-medium · 5-shot LogiQA, LogiQA2, SciQ, PiQA | mean accuracy: 39.4 -> 40.9 | Helped |
| Use Quadratic reweighting instead of uniform (ablation) | GPT2-medium · 5-shot LogiQA, LogiQA2, SciQ, PiQA | mean accuracy: 39.4 -> 39.3 | Hurt |
| Use Extremes reweighting instead of uniform (ablation) | GPT2-medium · 5-shot LogiQA, LogiQA2, SciQ, PiQA | mean accuracy: 39.4 -> 39.8 | Helped |
| Add LinUpper on top of DoGE domain reweighting | GPT2-medium non-uniform domain sampling · 5-shot LogiQA, LogiQA2, SciQ, PiQA | mean accuracy: 40.8 -> 41.5 | Helped |
| Add LinUpper on top of DoReMi domain reweighting | GPT2-medium non-uniform domain sampling · 5-shot LogiQA, LogiQA2, SciQ, PiQA | mean accuracy: 41.0 -> 41.5 | Helped |
| Add LinUpper loss-based instance reweighting over uniform sampling | Llama 1.4B pretraining · FineWeb subset, LUR 13 tasks | mean accuracy: 48.51 -> 49.16 (+0.66) | Helped |
| Add LinUpper loss-based instance reweighting over uniform sampling | Llama 1.4B pretraining · FineWeb subset, QA 6 tasks | mean accuracy: 42.35 -> 44.07 (+1.72) | Helped |
| Add LinUpper loss-based instance reweighting over uniform sampling | Llama 7B pretraining · FineWeb subset, LUR 13 tasks | mean accuracy: 50.54 -> 51.98 (+1.44) | Helped |
Surprising findings
- Quadratic and Extremes weighting, which up-weight some low-loss samples, underperform uniform sampling on most GPT2 perplexity results, while LinUpper consistently helps.
- On the 7B model, LinUpper drops ARC Challenge by 2.39 points and TinyWinoGrande by 4.88 points, showing mixed per-task effects.
Limitations
- Gains on large models are modest (around 1 point average) and per-task results are mixed, with some tasks decreasing.
- Experiments with GPT2 models use only 5-shot QA tasks since other benchmarks were at random at that scale; single-run variance is not reported.
- Theory covers convex losses and interpolation; non-convex analysis is deferred and the connection to LLM losses is approximate.
Experimental setup
Datasets: SlimPajama-6B FineWeb 15T LogiQA LogiQA 2 SciQ PiQA BoolQ SocialIQA TriviaQA LM Eval Harness tasks
Benchmarks: LogiQA LogiQA 2 SciQ PiQA ARC Challenge ARC Easy COPA MMLU MNLI WinoGrande BoolQ SocialIQA
Metrics: perplexity accuracy test loss
Baselines: Uniform sampling DoGE DoReMi DRO-KL
Builds on: DoReMi DoGE SlimPajama DRO-KL
Closest papers by meaning
- Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.93)
- Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning (ICML 2025 · similarity 0.93)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.91)
- Escaping Collapse: The Strength of Weak Data for Large Language Model Training (NeurIPS 2025 · similarity 0.90)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.90)
- Importance Weighting Can Help Large Language Models Self-Improve (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.