Research map / Large language models / Reasoning and chain of thought
AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaSTaR, an adaptive data-sampling extension of STaR/RFT that prioritizes under-trained, harder observations (diversity) while using a training-accuracy-based curriculum to temper difficulty, reducing training FLOPs while improving accuracy.
Problem
Standard STaR/RFT samples training observations randomly, causing persistent over-training on solved examples and under-training on hard ones, which wastes compute and slows convergence; sampling harder examples also raises false-positive (flawed CoT) rates.
Approach
Replace random re-sampling with a priority heap ordered by last-sampled iteration and a win-rate difficulty estimate (reusing STaR's K samples at no extra cost), and regularize difficulty with a curriculum that only updates a fraction floor(m*alpha^2) of popped observations, where alpha is training accuracy, so easy observations are repeated when the model is weak.
Key results
- AdaSTaR achieves the best test accuracy on 6 of 6 benchmarks with Llama 3.2 3B, e.g., 73.8% on ARC-C vs 73.2% for STaR-Acc and 78.0% on CQA.
- Relative to the strongest accuracy baseline, AdaSTaR reduces training PFLOPs by an average of 58.6% (range 19.2% to 93.7%).
- On Qwen 2.5 3B and Gemma 7B, AdaSTaR achieves best accuracy in 4 of 5 additional dataset settings (e.g., Gemma 7B ARC-C 85.4%, tied with STaR-Acc, with 16.2% fewer PFLOPs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdaSTaR adaptive sampling instead of the strongest accuracy baseline | Llama 3.2 3B · ARC-C | accuracy (%): 73.2 -> 73.8 | Helped |
| Use AdaSTaR adaptive sampling instead of the strongest accuracy baseline | Llama 3.2 3B · CQA | accuracy (%): 76.0 -> 78.0 | Helped |
| Use AdaSTaR adaptive sampling instead of the strongest accuracy baseline | Llama 3.2 3B · CLadder 1.5 | accuracy (%): 94.2 -> 95.6 | Helped |
| Use AdaSTaR adaptive sampling instead of the strongest accuracy baseline | Llama 3.2 3B · ANLI | accuracy (%): 64.8 -> 66.8 | Helped |
| Use AdaSTaR adaptive sampling instead of the strongest accuracy baseline | Qwen 2.5 3B · GSM8K | accuracy (%): 77.0 -> 77.0 | No clear effect |
| Use AdaSTaR adaptive sampling instead of the strongest accuracy baseline | Llama 3.2 3B · SVAMP | accuracy (%): 75.0 -> 75.5 | Helped |
| Use AdaSTaR instead of the strongest accuracy baseline | Llama 3.2 3B · ARC-C | PFLOPs: 639.8 -> 174.4 | Helped |
| Use AdaSTaR instead of the strongest accuracy baseline | Llama 3.2 3B · CQA | PFLOPs: 1158.3 -> 779.3 | Helped |
| Use AdaSTaR instead of the strongest accuracy baseline | Llama 3.2 3B · CLadder 1.5 | PFLOPs: 9663.0 -> 3610.0 | Helped |
| Use AdaSTaR instead of the strongest accuracy baseline | Llama 3.2 3B · ANLI | PFLOPs: 3528.4 -> 1340.9 | Helped |
| Use AdaSTaR instead of the strongest accuracy baseline | Qwen 2.5 3B · GSM8K | PFLOPs: 305.2 -> 19.3 | Helped |
| Use AdaSTaR instead of the strongest accuracy baseline | Llama 3.2 3B · SVAMP | PFLOPs: 229.3 -> 65.7 | Helped |
Surprising findings
- Removing the curriculum component (AdaD alone) sharply lowers accuracy (e.g., CLadder 1.5 drops from 95.6% to 52.4%) despite reducing observation training-frequency variance.
- Llama 3.2 3B fails to self-improve via STaR-based methods on GSM8K, requiring a switch to Qwen 2.5 3B.
Limitations
- Introduces a new hyperparameter f(alpha)=alpha^2 that is not finely tuned.
- Experiments limited to models up to 7B parameters due to compute budget.
- Does not combine with PRM or inference-time verifiers; interaction with RL long-CoT pipelines left open.
Experimental setup
Datasets: ARC-C CommonsenseQA CLadder 1.5 ANLI (R1) GSM8K SVAMP
Benchmarks: ARC-C CommonsenseQA CLadder 1.5 ANLI GSM8K SVAMP
Metrics: test accuracy PFLOPs standard deviation of observation training frequency
Baselines: SFT SFT + 8-CoT SFT + 5-SC STaR STaR-Full STaR-Acc STaR-Acc-Full STaR-Acc-Full-K ReST-EM B-STaR
Builds on: STaR RFT ReST-EM B-STaR V-STaR
Closest papers by meaning
- B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners (ICLR 2025 · similarity 0.86)
- Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning (NeurIPS 2024 · similarity 0.86)
- Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling (ICLR 2025 · similarity 0.86)
- First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training (NeurIPS 2025 · similarity 0.85)
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.85)
- Reasoning with Sampling: Your Base Model is Smarter Than You Think (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.