Research map / Large language models / Reasoning and chain of thought

BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BLADE, a Hessian-free penalized bi-level data selection method that recovers an excess-loss criterion with a dynamic reference model, instantiated as randomized block-coordinate Frank-Wolfe for token-level online selection in continual pre-training.

Problem

Data selection for LLM training: influence-based bi-level methods need intractable inverse-Hessian computations, while excess-loss methods (e.g., RHO-1) use a static reference model that becomes misaligned with the evolving proxy model.

Approach

Reformulate the bi-level data selection objective as a penalized single-level problem via Lagrange multipliers, which yields an excess-loss scoring rule, but replace the fixed reference model with one periodically re-trained (and re-synchronized with the proxy) on validation plus training signal; optimize selection with memoryless randomized block-coordinate Frank-Wolfe, giving top-k token masks.

Key results

• On TinyLlama-1.1B math pre-training, BLADE averages 29.7 vs 25.7 for RHO-1 and 16.7 for Base-CT (9 math benchmarks).
• On Llama2-7B math pre-training, BLADE averages 43.8 vs 41.1 for RHO-1 and 33.8 for Base-CT.
• On general pre-training with TinyLlama-1.1B, BLADE averages 51.73% vs 50.58% for RHO-1, winning 7 of 8 tasks.

Surprising findings

• RHO-1's token score distribution collapses toward non-positive values as its static reference lags, while BLADE's dynamic reference keeps a positive right tail.
• BLADE's selection quality is insensitive to the reference update interval tau in {500,...,2000}.

Limitations

• Theoretical guarantees rely on strong assumptions (PL condition, bounded Hessian, Lipschitz Hessian) and a large penalty lambda that is in practice replaced by a moderate value.
• Computational overhead is higher than static RHO-1 and much higher than vanilla continual training (494.3 vs 314.6 min on TinyLlama).
• Evaluated only on continual pre-training with 5B candidate pools and base models, not full-scale pre-training.

Experimental setup

Datasets: OpenWebMath MetaMath MAmmoTH (Mammoth) OpenHermes-2.5 SlimPajama

Benchmarks: GSM8K MATH SVAMP ASDiv MAWPS TabMWP MathQA MMLU-STEM SAT MMLU HellaSwag OpenBookQA

Metrics: accuracy average accuracy wall-clock time (min)

Baselines: Base Base-CT Random MATES RHO-1

Builds on: RHO-1 MATES influence functions LESS penalty-based bi-level optimization

Closest papers by meaning

• LLM Data Selection and Utilization via Dynamic Bi-level Optimization (ICML 2025 · similarity 0.92)
• Efficient Data Selection at Scale via Influence Distillation (NeurIPS 2025 · similarity 0.91)
• GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration (NeurIPS 2024 · similarity 0.91)
• Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.90)
• One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.90)
• Data Selection via Optimal Control for Language Models (ICLR 2025 · similarity 0.89)