Research map / Reinforcement learning / RL from feedback

Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor’s Internal States

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes POISE, an RLVR algorithm where a lightweight linear probe on the policy's own hidden states and token entropy predicts a per-prompt baseline, used with a cross-rollout construction (two rollouts per prompt) to keep gradients unbiased.

Problem

RLVR for reasoning models needs reliable baselines for variance reduction, but GRPO's group-mean baseline needs many rollouts per prompt (reducing prompt diversity), and PPO's LLM-scale critic roughly doubles cost.

Approach

A linear probe reads prompt and reasoning hidden states plus token-entropy statistics from the policy's forward pass to predict expected reward, trained online; a cross-rollout scheme predicts each rollout's baseline from the other independent rollout's features so the baseline is independent of the response being updated.

Key results

• On Qwen3-4B, POISE averages 58.10 vs 56.75 for DAPO (m=8) across 17 benchmarks, with POISE using only 2 rollouts per prompt.
• On OLMo-3-7B-Instruct-DPO, POISE averages 49.92 vs 47.96 for DAPO and 48.32 for RLOO (m=2).
• In the math setting on Qwen3-4B, the probe reaches Pearson r = 0.838 and MAE 0.126 against Avg@8 targets, better than the reasoning-only, entropy-only, and prompt-only variants.

Surprising findings

• RLOO with only m=2 rollouts is competitive with DAPO at m=8, and even surpasses it on OLMo-3-7B-Instruct-DPO.
• Removing the cross-rollout construction (same-rollout probe) drops Avg@32 from 50.1 to 40.7 on the math setting, showing that using response-dependent features without cross-rollout hurts.

Limitations

• Value estimator predicts sequence-level value only; no token-level credit assignment.
• Evaluated only on verifiable rewards; non-verifiable rubric rewards untested.
• Experiments are limited to a fixed compute budget and do not test longer training horizons.

Experimental setup

Datasets: GURU DAPO-Math-17K Dolci-Think-RL-7B IF-RLVR (IFBench) DAPO-Math DeepScaleR AceCoder ToolDial

Benchmarks: MATH500 AIME 2025 HumanEval MBPP LiveCodeBench ARC-AGI-1 Zebra Puzzle CodeI/O CRUXEval GPQA Diamond SuperGPQA HiTab

Metrics: accuracy Avg@32 MAE Pearson correlation

Baselines: DAPO RLOO PPO with token-level critic

Builds on: PPO GRPO DAPO RLOO GURU

Closest papers by meaning

• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.90)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.89)
• Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards (ICML 2026 · similarity 0.89)
• Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.89)
• Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.89)
• ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.89)