Research map / Large language models / Reasoning and chain of thought

Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dynamics-Predictive Sampling (DPS), which models each prompt's solving state as an HMM and uses online Bayesian inference to pick likely partially-solved prompts before rollouts, cutting rollout cost in RL finetuning of LLMs/VLMs.

Problem

RL finetuning of reasoning models benefits from selecting informative (partially solved) prompts, but existing online selection such as Dynamic Sampling requires rollouts over large candidate batches, whose cost can exceed the finetuning itself.

Approach

Treat each prompt's solving state (unsolved / partially solved / fully solved) as a hidden state in an HMM with a learned, non-stationary (exponentially decayed) Dirichlet transition model; update beliefs online from historical rewards and select the top-B prompts by predicted probability of being partially solved, before any rollouts.

Key results

• On MATH with DeepSeek-R1-Distill-Qwen-7B, DPS averages 63.13 across math benchmarks vs 62.42 for DS (Oracle) and 59.31 for US, using 287k rollouts versus DS's 1147k.
• On Countdown with Qwen2.5-7B, DPS reaches 81.15 on CD-34 vs 81.26 for DS, with 246k rollouts vs DS's 1006k.
• DPS matches or slightly exceeds DS on MATH while typically using under 30% of DS's rollout budget and roughly half its runtime.

Surprising findings

• DPS slightly outperforms the oracle DS on MATH benchmarks despite selecting top-B prompts rather than sampling randomly from partially solved prompts.
• History Resampling underperforms uniform sampling on some tasks, which the authors attribute to its rigid epoch-level absorbing assumption.

Limitations

• Relies on binary correctness rewards to define solving states; extension to dense or process rewards is not evaluated.
• Uses a simple top-k selection strategy that may not be optimal.
• Hyperparameter decay ratio lambda is tuned per task (0.7 for MATH, 0.5 for Countdown and Geometry3k).

Experimental setup

Datasets: MATH Countdown-34 Countdown-4 Geometry3k AIME24 AMC23 MATH500 Minerva Math OlympiadBench ARC-c MMLU-Pro

Benchmarks: AIME24 AMC23 MATH500 Minerva Math OlympiadBench ARC-c MMLU-Pro CD-34 CD-4 Geometry3k test

Metrics: Pass@1 accuracy averaged over 16 completions rollouts runtime effective sample ratio prediction accuracy/precision/recall/F1

Baselines: Uniform Sampling Dynamic Sampling History Resampling

Builds on: GRPO Dynamic Sampling History Resampling DAPO verl

Closest papers by meaning

• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.93)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.91)
• Reasoning with Sampling: Your Base Model is Smarter Than You Think (ICLR 2026 · similarity 0.91)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.91)
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening (ICML 2026 · similarity 0.91)