Research map / Large language models / Reasoning and chain of thought

Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AttnRL, a process-supervised RL framework for reasoning LLMs that branches MC rollouts at high attention-score (FCI) steps, adaptively samples problems and batches, and uses one-step off-policy sampling to cut sampling cost.

Problem

Process-supervised RL for reasoning LLMs suffers from inefficient exploration (poor branching positions, uniform sampling across prompts) and expensive two-stage Monte Carlo sampling per update.

Approach

Uses attention-based Forward Context Influence (FCI) scores to choose step-level branching points for tree-based MC advantage estimation, filters easy problems via FCI, scales tree count by difficulty, keeps batches valid with adaptive batch sizing, and overlaps initial sampling of the next batch with MC sampling of the current one.

Key results

• AttnRL on DS-R1-Distill-Qwen-1.5B reaches 57.2 average across six benchmarks vs 55.1 for TreeRL and 55.0 for GRPO, with 500 steps vs 800.
• On DS-R1-Distill-Qwen-7B, AttnRL averages 68.7 vs 66.9 for TreeRL and 66.3 for GRPO.
• Attention-based branching alone improves TreeRL average from 55.1 to 56.3 at 1.5B scale.

Surprising findings

• Disrupting attention at high-FCI steps degrades accuracy more than disrupting high-entropy steps, and earlier disruptions hurt more than later ones.

Limitations

• Evaluated only on mathematical reasoning benchmarks
• Only two distilled Qwen base models (1.5B and 7B) tested
• Gains over TreeRL are modest (about 1.8 points average at 1.5B)

Experimental setup

Datasets: DeepScaleR-Preview-Dataset

Benchmarks: AIME24 AIME25 AMC23 MATH-500 Minerva Math OlympiadBench

Metrics: Pass@1 Pass@K

Baselines: GRPO TreeRL DeepScaleR-Preview-1.5B

Builds on: TreeRL GRPO DAPO DeepScaleR

Closest papers by meaning

• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.92)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.91)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
• Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning (ICML 2024 · similarity 0.90)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.90)
• Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning (ICLR 2026 · similarity 0.89)