Research map / Multi-agent systems and games / LLM agents
When Instinct Guides and Insight Grounds: Staged RL Training for LLM Agents
Zijing Zhang, Boning Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a staged RL training approach for LLM agents in interactive environments, with the abstract only partially available; details beyond the abstract are unknown.
Problem
End-to-end RL training of LLM agents for dynamic, interactive decision-making is hard to optimize efficiently, and existing RL-based agent paradigms organize interactions as reasoning followed by action.
Approach
Staged RL training for LLM agents, framed around an instinct (fast action) versus insight (grounded reasoning) distinction; the abstract is truncated so the exact mechanism is not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add action-first exploration stage (stage 1) before reasoning-first training (ablation) | ActRe (Qwen2.5-1.5B) · ALFWorld | success rate: 0.73 -> 0.83 | Improved |
| add reasoning-first alignment stage (stage 2) (ablation) | ActRe (Qwen2.5-1.5B) · ALFWorld | success rate: 0.78 -> 0.83 | Improved |
| raise sampling temperature on key action tokens (ablation) | ActRe (Qwen2.5-1.5B) · ALFWorld | success rate: 0.81 -> 0.83 | Improved |
| add action-first exploration stage (ablation) | ActRe (Qwen2.5-1.5B) · WebShop | success rate: 0.57 -> 0.64 | Improved |
| add reasoning-first alignment stage (ablation) | ActRe (Qwen2.5-1.5B) · WebShop | success rate: 0.59 -> 0.64 | Improved |
| ActRe two-stage training instead of vanilla GRPO | GRPO (Qwen2.5-7B) · ALFWorld | success rate: 0.78 -> 0.86 | Improved |
| use higher sampling temperature during rollout | GRPO (Qwen2.5-1.5B) · ALFWorld | success rate: 0.73 -> 0.75 | No clear effect |
| use higher sampling temperature during rollout | GRPO (Qwen2.5-7B) · ALFWorld | success rate: 0.78 -> 0.80 | Mixed |
| add entropy regularization to the loss | GRPO (Qwen2.5-7B) · ALFWorld | success rate: 0.78 -> 0.82 | Improved |
| remove explicit reasoning (no think) in RL training (ablation) | GRPO (Qwen2.5-7B) · ALFWorld | pass rate: 77.6 -> 86.8 | Improved |
| remove explicit reasoning (no think) in SFT (ablation) | SFT (Qwen2.5-7B) · ALFWorld | pass rate: 63.8 -> 46.4 | Hurt |
| remove explicit reasoning (no think) in PPO (ablation) | PPO (Qwen2.5-1.5B) · ALFWorld | pass rate: 64.6 -> 61.4 | Hurt |
Limitations
- Only the abstract is available; full method and results could not be verified.
Closest papers by meaning
- Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.92)
- LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.91)
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.91)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.91)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.91)
- Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
