Research map / Large language models / Instruction and fine-tuning
Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DOTS, an attention-based online data selection that prioritizes moderate adaptive-difficulty questions with rollouts only on a reference set, plus Rollout Replay with a FIFO buffer and modified GRPO loss, cutting LLM RL fine-tuning time by 23–62%.
Problem
RL fine-tuning of LLMs for reasoning is resource-intensive, and data efficiency (number of steps and per-step cost) has been largely overlooked.
Approach
Estimate each question's adaptive difficulty (policy failure rate) by attention-weighted averaging over a small rollout-evaluated reference set, select questions with predicted difficulty near 0.5 online, and reuse recent non-degenerate rollouts from a FIFO buffer with importance-sampling-corrected GRPO loss.
Key results
- DOTS+RR reduces total RL fine-tuning time by 23%–62% across six LLM-dataset combinations, averaging 40.7% reduction, matching original GRPO accuracy.
- Rollout replay (fresh fraction 0.5) cuts per-step time by 11%–13%.
- Adaptive difficulty prediction achieves Pearson correlation of 0.70–0.78 with ground truth across settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prioritize questions with predicted adaptive difficulty near 0.5 (DOTS) | GRPO · Six LLM–dataset combinations | training steps to match original GRPO at 60 steps: 13.33% to 56.67% steps saved | Helped |
| reuse recent rollouts from a FIFO buffer (RR) | GRPO with DOTS · Six LLM–dataset combinations | per-step training time: 11%–13% reduction in per-step training time | Helped |
| combine difficulty-targeted online data selection and rollout replay | GRPO · Six LLM–dataset combinations | total training time: average of 40.7%; 61.65% on Qwen2.5-3B trained on DeepMath | Helped |
| add trained adapter layers and calibration instead of using off-the-shelf embeddings (ablation) | Adaptive difficulty prediction framework · Qwen2.5-Math-1.5B on MATH | Pearson correlation (ρ): 0.2682 ± 0.0207 -> 0.7843 ± 0.0243 | Helped |
| add trained adapter layers and calibration instead of using off-the-shelf embeddings (ablation) | Adaptive difficulty prediction framework · Qwen2.5-Math-1.5B on DeepScaleR | Pearson correlation (ρ): 0.2064 ± 0.0518 -> 0.7244 ± 0.0318 | Helped |
| add trained adapter layers and calibration instead of using off-the-shelf embeddings (ablation) | Adaptive difficulty prediction framework · Qwen2.5-Math-1.5B on ORZ | Pearson correlation (ρ): 0.1598 ± 0.0266 -> 0.7153 ± 0.0257 | Helped |
| add trained adapter layers and calibration instead of using off-the-shelf embeddings (ablation) | Adaptive difficulty prediction framework · Qwen2.5-3B on DeepScaleR | Pearson correlation (ρ): 0.2688 ± 0.0369 -> 0.7789 ± 0.0191 | Helped |
| add trained adapter layers and calibration instead of using off-the-shelf embeddings (ablation) | Adaptive difficulty prediction framework · Qwen2.5-3B on DeepMath | Pearson correlation (ρ): 0.0671 ± 0.0168 -> 0.7029 ± 0.0082 | Helped |
| add trained adapter layers and calibration instead of using off-the-shelf embeddings (ablation) | Adaptive difficulty prediction framework · Qwen2.5-Math-7B on DeepScaleR | Pearson correlation (ρ): 0.1983 ± 0.0254 -> 0.7076 ± 0.0195 | Helped |
| use a reference set size of 128 instead of 256 (ablation) | DOTS · DeepScaleR with Qwen2.5-Math-1.5B and Qwen2.5-3B | RL performance | No clear effect |
| apply DOTS and RR outside mathematics | GRPO on Qwen2.5-3B · Science subsets of MMLU (SCP-25K) | accuracy | Helped |
Surprising findings
- Off-the-shelf frozen embeddings give Pearson correlation as low as 0.07–0.27, far below the trained adapter (0.70–0.78).
- Rollouts for reference questions near difficulty 0.5 are reused in training, reducing rollout cost by 4–12% per step.
Limitations
- Reference set is randomly sampled each step, so prediction quality depends on reference set quality.
- Replay strategy is simple random replay of non-degenerate groups; no prioritized replay explored.
- Evaluation excludes AIME24 and AMC23 due to high variance; results mostly on math with one science experiment.
Experimental setup
Datasets: MATH DeepScaleR-40K Open-Reasoner-Zero-57K DeepMath-103K SCP-25K GSM8K MATH500 Minerva Math OlympiadBench MMLU
Benchmarks: GSM8K MATH500 Minerva Math OlympiadBench MMLU science subsets
Metrics: average accuracy Pearson correlation of predicted vs ground-truth adaptive di steps saved time saved per step total training time saved
Baselines: GRPO external-difficulty curriculum (GPT-4o-mini annotated) LIMR
Builds on: GRPO PPO experience replay Platt scaling DAPO
Closest papers by meaning
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.89)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.89)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.89)
- Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.89)
- Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.