Research map / Large language models / Reasoning and chain of thought

Self-Aligned Reward: Towards Effective and Efficient Reasoners

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes self-aligned reward (SAR), a perplexity-drop signal (ppl(a) vs ppl(a|q)) combined with verifiable rewards in PPO/GRPO, to improve both accuracy and conciseness of LLM reasoning.

Problem

RL with verifiable rewards gives only binary correctness feedback, causing verbose or redundant reasoning, while length penalties trade off accuracy.

Approach

SAR is the relative perplexity drop of an answer when the query is included versus standalone, computed from log-probs already available in RL training, and added to the verifiable reward. It favors concise, query-specific answers and penalizes memorized or verbose ones.

Key results

• SA-GRPO on Qwen3-4B reaches 71.41% average accuracy vs 69.07% for GRPO, with average length 894 vs 1031 tokens.
• SA-GRPO on Phi-3.5-mini gives 46.19% average accuracy and 368 average length vs GRPO's 45.91% and 448 tokens.
• SA-GRPO reports an average of at least 4% accuracy gain and 30% length reduction over GRPO, per the paper's claim (the table shows smaller accuracy gains for some models).

Surprising findings

• Using only the self-aligned reward without verifiable reward collapses accuracy (20.96% average on Qwen3-4B) into very short, shallow answers.
• Entropy minimization as an internal reward yields limited accuracy gains with a large efficiency drop.

Limitations

• Gains on vision-language models are less consistent; SA-GRPO does not transfer as strongly to Qwen2-2B-VL.
• Reported accuracy and length improvements vary across models and benchmarks, e.g., some cases show SA-GRPO with longer length than length-based baselines.
• Answer-type analysis relies on GPT-4o annotation and a 200-question sample.

Experimental setup

Datasets: GSM8k MATH NuminaMath 1.5

Benchmarks: GSM8k GSM-symbolic MATH NuminaMath AIME 1983-2024 LogicBench ProntoQA

Metrics: accuracy average response length accuracy-efficiency trade-off score (AES) advantage values GPU hours

Baselines: PPO GRPO GRPO-O1 GRPO-ER entropy minimization

Builds on: PPO GRPO Dr. GRPO O1-pruner Efficient Reasoner

Closest papers by meaning

• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.91)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.91)
• Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.91)
• Step-wise Rubric Rewards for LLM Reasoning (NeurIPS 2026 · similarity 0.91)
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.91)
• Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning (ICML 2026 · similarity 0.90)