Research map / Large language models / Reasoning and chain of thought

Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GVM-RAFT, a prompt-specific dynamic sample allocation for rejection-sampling fine-tuning (RAFT++) and GRPO that minimizes stochastic gradient variance under a fixed inference budget, showing faster convergence on math reasoning with Qwen2.5-Math models.

Problem

Uniform best-of-n sampling in RAFT-style CoT training ignores prompt difficulty and gradient magnitude, giving inefficient stochastic gradient estimates of the EM/ELBO objective under a fixed inference budget.

Approach

Allocate per-prompt sampling budget proportional to G_i / sqrt(p_i + alpha/p_i^(beta-1)), where p_i is the acceptance rate and G_i a gradient-norm (Lipschitz) estimate, derived by minimizing an upper bound on gradient variance; apply this to RAFT++ and also to GRPO.

Key results

• GVM-RAFT++ reaches about 2x speedup (N'=8) and 4x speedup (N'=32) in update steps over vanilla RAFT++ on Qwen2.5-Math-1.5B.
• On Qwen2.5-Math-1.5B, GVM-GRPO attains a 5-benchmark average of 40.42 versus 38.11 for GRPO and 36.42 for RAFT++.
• On Qwen2.5-Math-7B the gains are small: GVM-GRPO 49.86 vs GRPO 49.07, GVM-RAFT++ 49.13 vs RAFT++ 49.06.

Measured effects reported in the paper

Surprising findings

• Using a larger first-stage sample size N' made GVM-RAFT++ training rewards lower than RAFT++, attributed to biasing budget toward harder prompts.
• Clip-higher was reported to hurt GRPO, so it was omitted for GRPO training.

Limitations

• Experiments only on Qwen2.5-Math base models; generalization to other base models untested.
• Theoretical guarantees rely on smoothness/convexity assumptions that may not hold for LLMs.
• Gains on the 7B model are marginal relative to baselines, mainly a convergence-speed improvement.

Experimental setup

Datasets: Numina-Math

Benchmarks: Math500 Minerva Math Olympiad Bench AIME24 AMC23

Metrics: average accuracy @ 8 5-benchmark average

Baselines: RAFT++ GRPO Base model

Builds on: RAFT GRPO PPO TRICE

Closest papers by meaning

• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
• Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning (NeurIPS 2025 · similarity 0.89)
• Reasoning with Sampling: Your Base Model is Smarter Than You Think (ICLR 2026 · similarity 0.89)
• Optimizing Anytime Reasoning via Budget Relative Policy Optimization (NeurIPS 2025 · similarity 0.88)
• Metastable Dynamics of Chain-of-Thought Reasoning: Provable Benefits of Search, RL and Distillation (ICML 2025 · similarity 0.88)
• Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.88)