Research map / Large language models / Reasoning and chain of thought
Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Satori, a 7B LLM trained via small-scale format tuning on COAT (chain-of-action-thought) demonstrations followed by large-scale RL with restart-and-explore and reflection bonuses, enabling single-model autoregressive search.
Problem
Single LLMs rely on external verifiers or two-player search for test-time scaling; the paper asks whether search capabilities (self-reflection and exploring alternatives) can be internalized into one LLM's autoregressive reasoning.
Approach
Introduces Chain-of-Action-Thought (COAT) with meta-action tokens (continue, reflect, explore) and a two-stage recipe: format tuning on 10K synthetic multi-agent demonstrations, then PPO-based RL with Restart-and-Explore (restarting from correct/incorrect partial trajectories), reflection bonuses, and an outcome reward model, plus iterative self-improvement via SFT distillation.
Key results
- Satori-Qwen-7B reaches 85.6 on MATH500 and 20.0 on AIME2024, outperforming Qwen-2.5-Math-7B-Instruct (83.6, 16.7) and averaging 62.6 across five math benchmarks; round 2 averages 64.4.
- On out-of-domain benchmarks Satori-Qwen-7B averages 60.4 vs 52.5 for Qwen-2.5-Math-7B-Instruct, with BoardgameQA 61.8 vs 51.3.
- A 7B model trained with 300K FT only (no RL) scores lower than Satori-Qwen-7B (10K FT + 300K RL), e.g., MATH500 78.2 vs 85.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COAT instead of CoT, adding reflect and explore actions (ablation) | Qwen-2.5-Math-7B trained with SFT+RL · GSM8K | accuracy: 93.1 -> 93.2 | No clear effect |
| Use COAT instead of CoT, adding reflect and explore actions (ablation) | Qwen-2.5-Math-7B trained with SFT+RL · MATH500 | accuracy: 84.4 -> 85.6 | Helped |
| Use COAT instead of CoT, adding reflect and explore actions (ablation) | Qwen-2.5-Math-7B trained with SFT+RL · OlympiadBench | accuracy: 42.7 -> 46.6 | Helped |
| Use COAT instead of CoT, adding reflect and explore actions (ablation) | Qwen-2.5-Math-7B trained with SFT+RL · AMC2023 | accuracy: 60.0 -> 67.5 | Helped |
| Use COAT instead of CoT, adding reflect and explore actions (ablation) | Qwen-2.5-Math-7B trained with SFT+RL · AIME2024 | accuracy: 10.0 -> 20.0 | Helped |
| Train with a second round of RL and iterative self-improvement (ablation) | Satori-Qwen-7B · MATH500 | accuracy: 85.6 -> 83.6 | No clear effect |
| Train with a second round of RL and iterative self-improvement (ablation) | Satori-Qwen-7B · OlympiadBench | accuracy: 46.6 -> 48.5 | Helped |
| Train with a second round of RL and iterative self-improvement (ablation) | Satori-Qwen-7B · BGQA | accuracy: 61.8 -> 58.5 | Hurt |
| Use 10K FT data plus 300K RL data instead of 300K FT data without RL (ablation) | Satori-Qwen-7B · MATH500 | accuracy: 78.2 -> 85.6 | Helped |
| Use 10K FT data plus 300K RL data instead of 300K FT data without RL (ablation) | Satori-Qwen-7B · BGQA | accuracy: 50.5 -> 61.8 | Helped |
| Use RL-trained Satori-generated data instead of format-tuning data for distillation (ablation) | Llama-3.1-8B and Granite-3.1-8B · Math benchmarks average | accuracy | Helped |
Surprising findings
- Large-scale format tuning with 300K demonstrations underperforms small-scale FT plus RL, especially out-of-domain (e.g., CRUXEval 29.5 vs 42.5).
- Self-correction rates after RL are higher than after format tuning alone, with positive correction on MATH500 rising from 20.6% to 61.0%.
Limitations
- Results on competition benchmarks are based on small test sets (AMC2023 40 problems, AIME2024 30 problems), so differences of a few points may be noisy.
- The COAT format is restricted to a few hand-designed meta-actions (continue, reflect, explore) and the paper leaves a broader action set to future work.
- Theoretical justification for the restart-and-explore initial-state distribution is deferred to future work.
Experimental setup
Datasets: OpenMathInstruct-2 NuminaMath-CoT MATH GSM8K MATH500 OlympiadBench AMC2023 AIME2024 FOLIO BoardgameQA CRUXEval StrategyQA
Benchmarks: GSM8K MATH500 OlympiadBench AMC2023 AIME2024 FOLIO BoardgameQA CRUXEval StrategyQA TableBench MMLU-Pro STEM
Metrics: zero-shot pass@1 accuracy self-correction positive/negative rate
Baselines: Qwen-2.5-Math-7B-Instruct Qwen-2.5-7B-Instruct Llama-3.1-8B-Instruct OpenMath2-Llama3.1-8B NuminaMath-7B-CoT Llama-3.1-70B-Instruct OpenMath2-Llama3.1-70B QwQ-32B-Preview o1-preview GPT-4o Qwen-7B-CoT
Builds on: PPO Go-Explore chain-of-thought STaR-style self-improvement SoS
Closest papers by meaning
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.92)
- ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.91)
- Hybrid Latent Reasoning via Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.90)
- Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.