Research map / Large language models / Reasoning and chain of thought

Step-wise Rubric Rewards for LLM Reasoning

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SRaR, an RLVR framework that attributes rubric items (SUGGEST/PITFALL/BONUS) to individual reasoning steps, normalizes step rewards across rollouts, and adds them via a decoupled advantage on top of the outcome advantage, plus a 16K-problem contrastive-distilled rubric corpus.

Problem

Outcome-only RLVR gives no supervision over intermediate steps, and existing rubric-as-reward methods (RaR) collapse per-step verdicts into one response-level scalar, which mis-credits steps and invites rubric-gaming behaviors such as self-correction looping.

Approach

Attribute each rubric item to the reasoning step it judges, normalize per-step rubric rewards across rollouts, broadcast them only to that step's tokens, and keep them in a separate advantage term (decoupled from the outcome GRPO baseline) so rubric noise cannot inflate the group baseline.

Key results

• SRaR reaches 72.53 average on Qwen3-8B-Non-Thinking, +3.57 over RaR (68.96).
• On Qwen3-32B-Non-Thinking SRaR averages 75.81, +2.75 over RaR (73.06).
• Faithful Reasoning Rate on AIME 2025 rises from 34.5% (RaR) to 46.7%, and looping rate falls from 48.1% to 26.5%.

Surprising findings

• Removing cross-rollout step normalization collapses training to 56.76 average, below the base model on several benchmarks, with the model collapsing reasoning into a single step.
• Only 18.2% of steps in answer-correct responses are wrong, yet 49.9% of steps in answer-incorrect responses are correct, motivating step-level rewards.

Limitations

• Depends on external LLM judge for satisfaction and step attribution; judge errors propagate into rewards.
• Evaluated only on mathematical reasoning; other domains untested.
• Requires explicit step headers via format prompt and relies on GPT-5 for rubric generation.

Experimental setup

Datasets: DAPO math dataset AIME 2024 AIME 2025 AMC 2023 MATH500 Minerva Math OlympiadBench

Benchmarks: AIME 2024 AIME 2025 AMC 2023 MATH500 Minerva Math OlympiadBench

Metrics: Avg@32 accuracy average accuracy Faithful Reasoning Rate Misaligned Trajectory Rate looping rate step accuracy

Baselines: GRPO DAPO RaR GRPO-VPS RGR-GRPO

Builds on: GRPO DAPO Rubrics as Rewards (RaR) RGR-GRPO GRPO-VPS

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR (NeurIPS 2026 · similarity 0.91)
• RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.91)
• Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.91)
• RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning (ICML 2026 · similarity 0.91)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.91)