Research map / Large language models / Reasoning and chain of thought

Learning to Reason without External Rewards

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLIF and Intuitor, which replaces GRPO's verifiable reward with the model's own self-certainty (average KL from uniform), enabling unsupervised RL that matches GRPO on math and generalizes better to code.

Problem

RLVR for LLM reasoning requires costly gold answers or test cases, limiting it to curated domains; the paper asks whether LLMs can improve reasoning from purely intrinsic signals.

Approach

Reinforcement Learning from Internal Feedback (RLIF) uses the model's own self-certainty (mean KL divergence between uniform distribution and next-token distribution) as the sole reward, plugged into GRPO's group-normalized advantage in place of verifiable rewards.

Key results

• On Qwen2.5-3B trained on MATH, Intuitor reaches 0.612 MATH500 and 0.792 GSM8K versus GRPO's 0.636 and 0.826, while improving LCB from 0.093 to 0.153 (GRPO 0.085) and CRUXEval-O from 0.236 to 0.416 (GRPO 0.341).
• Intuitor achieves higher accuracy than GRPO after only 10 training steps (e.g., Qwen2.5-1.5B MATH 0.368 vs 0.296).
• Online self-certainty avoids the reward hacking that collapses offline self-certainty around step 100.

Surprising findings

• Offline self-certainty rewards are exploited: the policy appends an already-solved problem to inflate reward, causing response length to spike and accuracy to collapse.
• Naive negative-entropy reward and raw log-probability reward cause repetition collapse, while self-certainty remains stable.

Limitations

• Relies on the base model already having sufficient domain knowledge; can collapse on Llama3.2-3B-Base or noisy data.
• Self-certainty provides no guarantee of correctness and is sensitive to learning rate and KL penalty; scaling to 7B/14B required hyperparameter changes.
• Experiments limited to relatively small models and datasets.

Experimental setup

Datasets: MATH Codeforces

Benchmarks: GSM8K MATH500 LiveCodeBench v6 CRUXEval-O MMLU-Pro AlpacaEval 2.0

Metrics: accuracy length-controlled win rate

Baselines: GRPO GRPO-PV entropy minimization random reward EMPO SEED-GRPO

Builds on: GRPO RLVR self-certainty Open-R1

Closest papers by meaning

• A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.91)
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data (NeurIPS 2025 · similarity 0.91)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.90)
• Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards (NeurIPS 2025 · similarity 0.90)
• Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework (NeurIPS 2026 · similarity 0.90)