Research map / Large language models / Reasoning and chain of thought

LaSeR: Reinforcement Learning with Last-Token Self-Rewarding

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LaSeR, which adds an MSE loss aligning a last-token self-rewarding score (log-prob of a pre-specified special token at the final response token, minus a constant) with verifier rewards, jointly training reasoning and self-verification in GRPO at near-zero extra inference cost.

Problem

RLVR gives no verification signal at test time, and prior self-verification methods require generating solutions and verifications sequentially, doubling inference cost.

Approach

Theoretically, the closed-form optimum of the self-verification RL objective makes the true verifier reward equal to the policy's log-probability ratio for a pre-specified token at the solution's last token (scaled by KL coefficient). Since the reference probability is near-constant, this reduces to an MSE loss on the policy's next-token log-prob minus a constant, added to RLVR, so reasoning and s

Key results

• LaSeR raises self-verification F1 on Open-Reasoner-Zero-7B to 77.6 average versus 38.8 for GRPO, and reasoning average accuracy to 45.5 vs 45.0.
• On Qwen2.5-7B-Base, LaSeR reaches 42.7 average reasoning accuracy (GRPO 41.8) and 79.6 self-verification F1 (GRPO 49.2).
• LaSeR's self-rewarding (7B) on Qwen2.5-7B-LaSeR generations reaches 79.6 average F1, outperforming 7B PRMs and matching the 72B reward model (80.0).

Surprising findings

• Self-rewarding loss was reported to degrade reasoning when using an SFT-style loss, while the MSE formulation caused little interference.
• Simplifying the reference log-probability to a constant gives comparable results while halving self-rewarding computation.

Limitations

• Self-verification F1 is lower on general reasoning tasks (65-70%), so self-rewarding advantages were disabled there.
• Relies on a pre-specified unused special token and empirically chosen hyperparameters (beta_v, alpha, tau).
• Self-verification performance on AIME benchmarks for OctoThinker models was not reported due to few correct solutions.

Experimental setup

Datasets: DeepMath-103K WebInstruct-verified

Benchmarks: MATH500 AMC23 AIME24 AIME25 OlympiadBench MMLU-Pro GPQA-Diamond

Metrics: Pass@1 accuracy self-verification F1 majority voting accuracy weighted majority voting accuracy

Baselines: GRPO Qwen2.5-Math-7B-PRM800K Qwen2.5-Math-PRM-7B Qwen2.5-Math-RM-72B

Builds on: GRPO RLVR DPO implicit reward verl

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.92)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.91)
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.91)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.91)
• Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.91)