Research map / Large language models / Reasoning and chain of thought

D²Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes D²Evo, a dual difficulty-aware self-evolving Questioner–Solver GRPO loop that mines mid-difficulty real anchors each iteration and trains on a hybrid of real and generated mid-difficulty math problems, reaching strong math results with under 2K real samples.

Problem

GRPO-style RL for LLM reasoning needs medium-difficulty samples, which are scarce in static datasets and become trivial as the model improves; prior self-play methods generate questions without anchors, ignore co-evolution, or mismatch difficulty.

Approach

Each iteration, a frozen Solver estimates difficulty of real data to select mid-difficulty anchors; a Questioner conditioned on these anchors is trained with GRPO under a band-shaped difficulty reward; generated questions are pseudo-labeled by Solver majority vote and filtered to mid-difficulty, and the Solver is trained on the hybrid buffer, closing a co-evolution loop.

Key results

• On Qwen3-8B-Base, D²Evo (Iter 3, 0.4K real samples) reaches 55.32 math average vs 54.34 for SPICE (20K docs) and 52.70 for Full Data (19K).
• On Qwen3-4B-Base, D²Evo Iter 3 reaches 51.35 math average vs 50.59 for SPICE and 49.28 for Full Data.
• Ablation removing the Questioner drops the Qwen3-8B math average from 55.32 to 51.40.

Surprising findings

• Removing the anchor-conditioned difficulty targeting (random anchors) costs about 2 points, indicating harder anchors introduced too early hurt training.
• The independently trained Questioner's performance declines or plateaus across iterations, while the co-evolved Questioner improves.

Limitations

• Relies on GPT-5.2 to double-check pseudo-labels and on GPT-4o for evaluation, adding API cost and dependence on external models.
• Gains on general reasoning are modest because training uses only math data.
• Inconsistent reported sample counts across tables (e.g., 1.4K vs 0.4K for the same model) reduce clarity of data-efficiency claims.

Experimental setup

Datasets: OpenRs-7K Math12K

Benchmarks: AMC Minerva MATH-500 GSM8K OlympiadBench AIME24 AIME25 MMLU-Pro SuperGPQA BBEH

Metrics: accuracy average accuracy question acceptance rate question difficulty

Baselines: Base Model Full Data R-Zero AZR SPICE

Builds on: GRPO R-Zero Absolute-Zero SPICE

Closest papers by meaning

• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
• QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation (ICLR 2026 · similarity 0.89)
• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• DyCon: Dynamic Reasoning Control via Evolving Difficulty Modeling (ICML 2026 · similarity 0.88)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.88)
• SuCo: Sufficiency-guided Continuous Adaptive Reasoning (ICML 2026 · similarity 0.88)