Research map / Reinforcement learning / RL from feedback

Not Only Where, But When: Temporal Scheduling for RLVR

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes temporally scheduling RLVR credit-allocation criteria (starting with targeted tokens and gradually expanding to all tokens), instantiated with trajectory-percentile suffix scheduling, which improves Qwen3-4B/8B reasoning and preserves policy entropy.

Problem

Existing RLVR credit allocation methods (token-level advantage reweighting and sparse token optimization) use stagnant allocation criteria throughout training, limiting resilient policy evolution and performance ceiling.

Approach

Introduce a temporal dimension: schedule the credit allocation criterion over training so it starts by emphasizing targeted tokens and gradually attenuates toward general full-token optimization; trajectory percentile (suffix-first, expanding to earlier tokens) serves as a simple criterion that works well with this schedule.

Key results

• TP-Schedule on Qwen3-4B improves over vanilla GRPO by 2.2 points on math benchmarks (65.3 to 67.5 avg) and 2.7 points on general reasoning (64.4 to 67.1 avg).
• On Qwen3-8B, TP-Schedule improves GRPO average from 68.0 to 68.8 on math and 69.5 to 70.5 on out-of-distribution benchmarks.
• Temporal scheduling applied to Forking Tokens raises math average from 64.8 to 66.2 on Qwen3-4B; scheduling preserves higher policy entropy than entropy-based advantage reweighting.

Surprising findings

• Prefix-first (earlier-to-later) scheduling yields unstable optimization, while suffix-first scheduling works well.
• Randomly selecting tokens for scheduling significantly corrupts policy gradients.

Limitations

• Gains are modest (around 1-3 points) and evaluated on a limited set of models and benchmarks.
• Scheduling hyperparameters (temporal range, schedule shape) are tuned with sensitivity ablations on Qwen3-4B only.

Experimental setup

Datasets: OpenMathReasoning DeepMath-103K

Benchmarks: AIME24 AIME25 HMMT25 Minerva OlympiadBench WinoGrande GPQA-Diamond MuSR

Metrics: Avg@16 Avg@4 accuracy

Baselines: GRPO GSPO Entropy Adv. Forking Tok.

Builds on: GRPO GSPO Entropy-based advantage reweighting Forking tokens VERL

Closest papers by meaning

• Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.90)
• You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories (NeurIPS 2026 · similarity 0.89)
• What are Key Factors for Updates in RL for LLM Reasoning? (NeurIPS 2026 · similarity 0.89)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.89)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.89)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training (NeurIPS 2025 · similarity 0.89)