Research map / Large language models / Reasoning and chain of thought

DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewards

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DUET, a dual-controlled token-budget layer over GRPO that allocates rollouts per prompt via cost-weighted Neyman allocation and aborts rollouts with a marker-gated rule, with importance reweighting.

Problem

RLVR training cost is dominated by rollout token generation; prior efficient methods control either which prompts get rollouts or how long rollouts run, but not both under one shared token budget.

Approach

Jointly allocates rollout counts per prompt (cost-weighted Neyman rule with an online variance surrogate) and stops rollouts with a marker-gated abort using quantile thresholds and epsilon-keep importance reweighting, both coupled through one Lagrange budget multiplier.

Key results

• On Qwen3-1.7B, DUET at 50% rollout budget reaches 62.1 MATH-500 and 2.51x speedup over full-budget GRPO (56.8 at full GRPO).
• At full budget DUET reaches 1.62x speedup with 61.6 MATH-500 and 84.7 GSM8K on Qwen3-1.7B.
• Ablation: allocator alone and abort alone each improve MATH-500 over GRPO; combined gives +5.7 on MATH-500 at 1.73x speedup.

Surprising findings

• The gap between DUET and the strongest baseline widens as the budget tightens rather than shrinking.
• Realized importance-weight surcharge from the abort correction stays near 1x in practice despite a theoretical bound near 99x at epsilon=0.01.

Limitations

• Results are single-seed, with AIME-2024 directional only (30 problems).
• Marker-gated abort requires a domain-specific answer marker, limiting open-ended tasks.
• Evaluated only on smaller models due to compute limits; theory relies on idealized assumptions (action-independent baseline, true lengths).

Experimental setup

Datasets: MATH

Benchmarks: MATH-500 GSM8K AIME-2024 HumanEval GPQA-Diamond

Metrics: mean@4 accuracy wall-clock speedup

Baselines: GRPO DAPO ARRoL VIP

Builds on: GRPO DAPO VIP ARRoL stratified Monte Carlo

Closest papers by meaning

• Optimizing Anytime Reasoning via Budget Relative Policy Optimization (NeurIPS 2025 · similarity 0.89)
• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.88)
• Not Only Where, But When: Temporal Scheduling for RLVR (NeurIPS 2026 · similarity 0.88)
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners (ICLR 2026 · similarity 0.88)
• Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.88)
• One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.87)