Research map / Large language models / Reasoning and chain of thought

Reducing Belief Deviation in Reinforcement Learning for Active Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes T^3, which truncates RL rollouts once an LLM agent's belief deviation stalls (detected via redundant queries or lack of hypothesis-space contraction), improving PPO/GRPO/GSPO on active reasoning tasks.

Problem

LLM agents in active reasoning suffer belief deviation, entering a belief-trap region where uninformative trajectory tails contaminate credit assignment in RL training.

Approach

Theory-grounded truncation rule: detect sustained lack of hypothesis-space contraction over a window and cut the rollout there, so credit goes to informative prefixes; plugs into PPO, GRPO, GSPO without changing the algorithm.

Key results

• GRPO with T^3 improves GuessNumbers EM by 30.1 points over vanilla GRPO; GSPO+T^3 reaches 99.74 EM on GN.
• GSPO+T^3 improves MovieRecommendation EM by 41.0 points over vanilla GSPO.
• PPO+T^3 improves CircuitDecoding EM by 16.2 points (61.67 to 77.83), with token cost reduced by up to 34%.

Surprising findings

• Random truncation with a tuned ratio also yields mild gains on CD, suggesting long uninformative tails hurt optimization even without a precise signal.
• Frontier reasoning models (o3-mini, Gemini-2.5-Pro) underperform RL-trained 7B models with T^3 on unbounded hypothesis-space tasks like SP and PE.

Limitations

• Truncation proxies are task-specific and require task structure (hypothesis sets, ground-truth preference in PE), limiting generality.
• Theoretical assumptions (update-error growth, bounded policy sensitivity) are not directly verifiable for LLM agents.
• Sensitive to threshold choice: overly aggressive truncation hurts performance.

Experimental setup

Datasets: GuessNumbers SituationPuzzles CircuitDecoding PreferenceEstimation MovieRecommendation AR-Bench Multi-Turn Puzzles

Benchmarks: GuessNumbers SituationPuzzles CircuitDecoding PreferenceEstimation MovieRecommendation

Metrics: exact match F1-word F1-char binary similarity

Baselines: o3-mini Gemini-2.5-Pro Qwen-2.5-7B-Instruct PPO GRPO GSPO

Builds on: PPO GRPO GSPO GAE POMDP

Closest papers by meaning

• On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents (ICML 2026 · similarity 0.88)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.88)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.88)
• Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.88)
• Native Reasoning Models: Training Language Models to Reason on Unverifiable Data (ICLR 2026 · similarity 0.88)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.88)