Research map / Large language models / Alignment and preferences

LLM Data Selection and Utilization via Dynamic Bi-level Optimization

ICML 2025 · Large language models › Alignment and preferences · approach: Active and token-level preference learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a Data Weighting Model (DWM) that reweights samples within each batch during LLM pretraining, learned via a multi-stage bi-level optimization that maximizes validation (LAMBADA) performance, improving randomly and carefully selected data.

Problem

Existing LLM data selection methods are static and training-agnostic, selecting data before training and using selected samples indiscriminately within batches, ignoring the dynamic data preference of the model and joint effects between samples.

Approach

A plug-and-play data weighting model that assigns per-sample weights within each batch (considering inter-sample interactions) and is updated across training stages by a bi-level objective that maximizes the validation performance of the LLM trained with those weights, with the weighting model transferable to other selection methods and model sizes.

Key results

• On 370M models with random data, RANDOM+DWM reaches 46.4 two-shot average vs 45.1 for RANDOM (and 48.5 for QuRating, 48.6 for QuRating+DWM at 370M).
• At 1.3B, QuRating+DWM improves two-shot average from 50.3 to 51.0 and RANDOM+DWM from 47.6 to 48.7 when weighting model transferred from 370M.
• Ablations show the dynamic multi-stage weighting outperforms static weighting from stage 1 or stage 4 and that using LAMBADA as validation beats held-out SlimPajama validation (46.4 vs 46.1 two-shot).

Measured effects reported in the paper

Surprising findings

• Using only the final-stage weighting model boosts early training but reduces diversity, leading to worse overall performance than dynamic weighting.
• The DWM gain for QuRating is minor at 370M but larger at 1.3B, which the authors attribute to model capacity absorbing high-quality reasoning data.

Limitations

• Gains are small on average (about 0.1 to 1.3 points) and mixed across tasks, with some tasks degrading.
• Performance drops in the stage-2 weighting model, so consistent improvement across all stages is not guaranteed.
• Only evaluated on LAMBADA as validation task and on small models up to 1.3B.

Experimental setup

Datasets: SlimPajama LAMBADA RedPajama

Benchmarks: ARC-easy ARC-challenge SciQ LogiQA BoolQ OpenbookQA HellaSwag PIQA WinoGrande

Metrics: accuracy normalized accuracy

Baselines: RANDOM DSIR QuRating RANDOM_60B

Builds on: DSIR QuRating DsDM MATES RegMix

Closest papers by meaning

• BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training (NeurIPS 2026 · similarity 0.92)
• Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.90)
• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.90)
• Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.89)
• Importance Weighting Can Help Large Language Models Self-Improve (AAAI 2025 · similarity 0.89)
• Data Selection via Optimal Control for Language Models (ICLR 2025 · similarity 0.89)