Research map / Large language models / Reasoning and chain of thought

Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CDAS, a GRPO rollout sampler that estimates problem difficulty from accumulated performance discrepancies and selects problems whose difficulty aligns with the model's competence, framed as a contractive fixed-point system.

Problem

Rollout sampling efficiency in RL for LLM math reasoning: single-step pass-rate difficulty estimates are unstable and biased, and existing schedulers (curriculum, prioritized, dynamic sampling) do not model alignment between model competence and problem difficulty.

Approach

Define difficulty as a running average of gaps between sigmoid-predicted success (from competence minus difficulty) and observed pass rate, define competence as the negative mean difficulty, select problems with smallest |competence - difficulty| using a symmetric split into harder and easier groups, and prove the coupled update is a contraction (factor 1/2) so it converges to a unique fixed point

Key results

• CDAS reaches 45.89% average accuracy at step 110 on Qwen2.5-7B, the best among sampling methods (Dynamic Sampling 45.52%, Random Sampling 44.09%).
• CDAS reduces training step time by 57.06% relative to Dynamic Sampling, which is reported as 2.33x slower than CDAS-level cost.
• On Qwen2.5-14B, CDAS reaches 47.22% average accuracy versus 46.62% for Dynamic Sampling and 46.28% for Random Sampling.

Surprising findings

• Dynamic Sampling achieves the best AIME24 score (12.19 at step 55 and 15.10 at step 110 tied with Prioritized Sampling), so CDAS is not uniformly best.
• Curriculum Sampling gives exactly the same numbers as Random Sampling at step 55, and its gains over random are small (+0.35 at step 110).

Limitations

• Warm-up requires one full epoch of uniform random sampling before adaptive selection begins.
• Gains over the strongest baseline (Dynamic Sampling) are small on several benchmarks and within run-to-run noise on AIME.
• Code generation experiment is limited to a single 100-step run and only compares RS, DS and CDAS.

Experimental setup

Datasets: MATH APPS TACO CodeContests

Benchmarks: AIME24 AIME25 MATH500 Minerva Math OlympiadBench GSM8K LiveCodeBench v5

Metrics: Avg@32 accuracy Pass@1 training step time

Baselines: Random Sampling Curriculum Sampling Prioritized Sampling Dynamic Sampling

Builds on: GRPO Dynamic Sampling (DAPO) Prioritized Sampling (Kimi k1.5) Curriculum learning SimpleRL-Zoo veRL

Closest papers by meaning

• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.90)
• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.89)
• Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards (ICML 2026 · similarity 0.89)
• Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.89)