Research map / Large language models / Reasoning and chain of thought

SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAGE, which reshapes the reverse-KL anchor distribution with an intrinsic-signal guide function (entropy-based branch amplification) to expand empirical support and improve both pass@1 and pass@k in RLVR for math reasoning.

Problem

RLVR with reverse-KL regularization improves pass@1 but often fails to improve pass@k because the policy stays anchored to the reference distribution and suppresses low-density reasoning modes; removing KL or using forward-KL trades off stability and efficiency.

Approach

Treat the reverse-KL anchor as a shapeable target: replace pi_ref with q*pi_ref in the KL term, where q is a fixed per-iteration guide built from token-level entropy (branch-level: 1 + gamma*(H_t - tau)^+) or surprisal/random perturbations, so the stationary policy is pi_ref*q*exp(r/beta) and low-density reasoning branches gain mass without removing the stabilizing anchor.

Key results

• GRPO+Branch averages pass@1 0.284 and pass@256 0.729 over AIME/AMC23/MATH-500, versus GRPO 0.266 and 0.714 (5 seeds).
• GRPO w/o KL reaches avg pass@1 0.258 and pass@256 0.701, below GRPO, despite higher training reward.
• On DeepSeek-R1-Distill-Qwen-7B, GRPO+Branch gets avg pass@1 0.452 vs 0.449 for GRPO and pass@64 0.774 vs 0.771.

Surprising findings

• Removing the reverse-KL term yields higher training reward but worse test pass@1 and pass@256, indicating reward overfitting.
• On the distilled R1 model, GRPO lowers pass@64 relative to the base model (0.771 vs 0.778), and GRPO+Branch recovers only partly.

Limitations

• SAGE only redistributes mass among trajectories already supported by the reference policy; zero-probability trajectories stay unreachable.
• Guide function is instantiated with simple intrinsic signals (entropy, surprisal) rather than learned or teacher-derived guides.
• Gains are modest (e.g., avg pass@1 0.284 vs 0.266) and many differences are within reported standard deviations.

Experimental setup

Datasets: OpenMathReasoning-mini DAPO-Math-17K-Processed

Benchmarks: AIME 2024-2025 AMC23 MATH-500 (level 5) Knights and Knaves

Metrics: pass@1 pass@256 pass@64 pass@k curves

Baselines: GRPO GRPO w/o KL GRPO + Forward KL GRPO + Random GRPO + Token DAPO (no KL) BNPO (no KL)

Builds on: GRPO DAPO BNPO reverse-KL RLVR analyses pseudo-KL divergence

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.92)
• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.92)
• Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs (ICLR 2026 · similarity 0.91)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.91)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.91)