Research map / Large language models / Reasoning and chain of thought

Plan Then Action: High-Level Planning Guidance Reinforcement Learning for LLM Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PTA-GRPO, a two-stage plan-then-reason framework: SFT on LLM-summarized high-level plans followed by GRPO with an analytic-plan reward based on plan-conditioned answer accuracy, improving math reasoning over GRPO/DAPO.

Problem

Token-level CoT decoding is locally greedy and lacks global planning, causing redundant or inaccurate reasoning; existing RLVR methods only reward final answers and do not evaluate plan quality.

Approach

Treat a concise high-level plan as an explicit optimization variable: SFT on plan-augmented CoT data, then GRPO with a reward that scores each plan by the softmax-normalized empirical accuracy of CoTs sampled under it, added to outcome and format rewards.

Key results

• On Qwen2.5-7B-Instruct, PTA-GRPO averages 58.46 across MATH500, AIME24, AIME25, AMC23 versus 54.81 for GRPO and 55.04 for DAPO.
• On Qwen3-14B, PTA-GRPO reaches 85.01 average versus 83.31 for DAPO and 83.08 for GRPO.
• Ablation on Qwen2.5-7B-Instruct: removing SFT drops the average to 46.33; removing the analytic reward drops it to 54.56.

Surprising findings

• Plans self-generated by the policy model itself yield nearly the same performance (57.37 vs 58.46 average) as plans from Qwen3-235B.
• Gains over GRPO on Qwen3-8B are small and some are not statistically significant (e.g., AMC23 p=0.41).

Limitations

• Gains over GRPO on larger models (Qwen3-8B, Qwen3-14B) are modest and sometimes not significant.
• Plan quality is only proxied by plan-conditioned accuracy; the theoretical bound relies on an assumed monotonicity condition on conditional entropy.
• Evaluation is concentrated on competition math; multimodal results are limited to one VLM and compared mainly to SRPO.

Experimental setup

Datasets: OpenThoughts DeepMath SRPO SFT/RL data

Benchmarks: MATH500 AIME24 AIME25 AMC23 MMMU-Pro MMMU EMMA MMK-12

Metrics: average accuracy over 64 runs Pass@k accuracy

Baselines: GRPO DAPO CPL Full-Step-DPO ORZ SRPO

Builds on: GRPO DAPO Chain-of-Thought SRPO

Closest papers by meaning

• Policy Guided Tree Search for Enhanced LLM Reasoning (ICML 2025 · similarity 0.92)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.91)
• Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search (ICML 2025 · similarity 0.89)
• Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning (ICLR 2026 · similarity 0.89)
• STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning (NeurIPS 2026 · similarity 0.89)
• Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.89)