Research map / Large language models / Reasoning and chain of thought

Reducing Credit Assignment Variance via Counterfactual Reasoning Paths

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IBPO, which pairs incorrect sampled reasoning trajectories with reference trajectories to derive implicit process-level shaping signals on top of GSPO under sparse terminal rewards, reporting gains on AIME25, LiveCodeBench, and HMMT25 with Qwen3 models.

Problem

Sparse terminal rewards in RL for multi-step LLM reasoning propagate credit uniformly across all intermediate decisions, causing high gradient variance, unstable training, and inefficient updates.

Approach

Treats differences among multiple on-policy trajectories for the same prompt as counterfactual approximations; an incorrect trajectory is compared against correct or other incorrect references (via a compare-and-correct operator) to produce a shaping reward and optional token-level gradient masks, injected into a sequence-level optimizer without step labels or a value model.

Key results

• GSPO+IBPO improves Qwen3-32B over GSPO by 8.2 (AIME25), 10.7 (LiveCodeBench), and 7.0 (HMMT25) points (85.3, 75.3, 62.6 vs 77.1, 64.6, 55.6).
• IBPO reaches a training reward of 0.75 with 0.63x the training compute of GSPO on Qwen3-Next-80B-A3B-Thinking.
• Ablation: test-time prompting only gives 78.2 on AIME25, versus 85.3 for full IBPO trained jointly.

Surprising findings

• Test-time-only correction with the same prompt gives only marginal gains over GSPO, suggesting the benefit depends on joint training.

Limitations

• Extra compute overhead from sampling and correcting multiple counterfactual trajectories per prompt.
• Systematic errors in counterfactual trajectories can weaken the comparison signal.
• Relies on verifiable terminal rewards; applicability to open-ended tasks with LLM judges is unverified.

Experimental setup

Datasets: AIME25 HMMT25 LiveCodeBench v6

Benchmarks: AIME25 LiveCodeBench HMMT25

Metrics: pass@1 accuracy training compute to reach reward threshold

Baselines: GSPO GSPO with prompt GSPO with SCoRe GSPO with Best-of-N

Builds on: GSPO GRPO SCoRe

Closest papers by meaning

• Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning (AAAI 2025 · similarity 0.89)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.88)
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.88)
• VinePPO: Refining Credit Assignment in RL Training of LLMs (ICML 2025 · similarity 0.88)
• Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning (AAAI 2026 · similarity 0.87)
• Optimizing Anytime Reasoning via Budget Relative Policy Optimization (NeurIPS 2025 · similarity 0.87)