Research map / Large language models / Reasoning and chain of thought

Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADAPT, an online data reweighting method that scales per-sample learning rates via similarity between training samples and a validation anchor set (using the model's own hidden states), evaluated on LoRA instruction tuning of Llama-2-7B and TinyLlama-120M pretraining on SlimPajama.

Problem

Offline data curation (selection and mixing) is decoupled from training, costs extra compute, overfits to validation benchmarks, and must be rerun when model or task changes.

Approach

Recast data curation as online sample-level loss reweighting that keeps the full dataset and scales each sample's gradient step by a sigmoid of its similarity to a validation anchor set, with anchor embeddings from the current model refreshed periodically.

Key results

• On MMLU-to-BBH cross-benchmark generalization, ADAPT reaches 44.8 vs 44.4 for full-dataset SFT and 38.7 for LESS (table).
• On TinyLlama-120M at 50B tokens, ADAPT achieves 38.00 average over all tasks vs 37.81 for Uniform and 37.97 for RegMix, with 33.73 on unseen tasks vs 31.98 for Uniform.
• At 50B tokens, validation perplexity on SlimPajama drops from 17.15 (Uniform) to 16.55 (ADAPT).

Surprising findings

• LinUpper, an online sample reweighting baseline, performs worse than Uniform sampling on average (37.03 vs 37.81).
• On pretraining downstream tasks, ADAPT's average gain over Uniform is small (0.19 points at 50B tokens), and RegMix and ADAPT-BM25 are close.

Limitations

• Pretraining gains over Uniform are small (about 0.19 average points at 50B tokens), and ADAPT is not best on many individual tasks.
• Requires a validation anchor set drawn from the target evaluation distribution, so it is not fully task-agnostic.
• Instruction-tuning results rely on a single MMLU-to-BBH transfer setting and small BBH test set (920 examples), with high variance.

Experimental setup

Datasets: Flan V2 CoT Dolly Open Assistant 1 SlimPajama MMLU validation set

Benchmarks: MMLU BBH ARC-C ARC-E COPA HellaSwag Lambada-S Lambada-O LogiQA MultiRC OpenBookQA PIQA

Metrics: accuracy exact match validation perplexity FLOPs

Baselines: BM25 Embedding LESS PPL Random Full Dataset SFT Uniform LinUpper DoReMi RegMix

Builds on: LESS DoReMi RegMix TinyLlama LoRA

Closest papers by meaning

• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.93)
• BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training (NeurIPS 2026 · similarity 0.90)
• Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.90)
• LLM Data Selection and Utilization via Dynamic Bi-level Optimization (ICML 2025 · similarity 0.90)
• Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning (ICML 2025 · similarity 0.90)
• Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.90)