Research map / Reinforcement learning / RL from feedback

Learning Agentic Policy from Action Guidance

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ActGuide-RL, which injects action trajectories from expert search agents as plan-style hints only as an adaptive fallback on zero-reward groups, then mixes guided and unguided rollouts in one GRPO-style update to internalize exploration gains into the unguided policy.

Problem

Agentic RL for LLMs stalls when the base policy cannot reach reward states (out-region tasks), giving zero group advantage; existing fixes need costly SFT cold starts.

Approach

Use cheap, abundant action-only trajectories as plan-style reference in the prompt, invoke guidance only as a minimal-intervention fallback (least guidance level that yields a success), and optimize guided and unguided rollouts jointly with mixed-policy importance ratios so gains transfer to the unguided policy at inference.

Key results

• On Qwen3-4B-Instruct, ActGuide-RL reaches 35.92 on GAIA vs 25.24 for zero RL (+10.68 pp) and 37.00 vs 18.00 on XBench (+19 pp).
• On WebWalkerQA with Qwen3-4B-Instruct, average rises from 12.06 (zero RL) to 39.85, a +27.79 pp gain.
• ActGuide-RL without cold start matches SFT+RL on in-domain benchmarks (e.g., GAIA 40.77 with SFT+ActGuide vs 36.89 SFT+RL).

Surprising findings

• Over-guidance inflates off-policy risk: stronger guidance levels increase log-ratio variance, so fixed heavy guidance underperforms adaptive minimal guidance.
• Injecting actions as a reference plan in the query prompt outperforms assistant-prefix and user-assistant message injection (Reward@1 85.7 vs 74.5 and 80.1).

Limitations

• Evaluated only on search-agent settings, which are stateless; generalization to GUI or other agentic environments is untested.
• Benefit depends on availability of action trajectories for training tasks; noisy actions at 20% degrade performance.
• Some benchmark results are below SFT+RL on certain metrics (e.g., XBench 17.00 for SFT+RL vs 25.00 with SFT+ActGuide).

Experimental setup

Datasets: ASearcher GAIA WebWalkerQA XBench BrowseComp-ZH GPQA TruthfulQA IFEval

Benchmarks: GAIA WebWalkerQA XBench BrowseComp-ZH GPQA TruthfulQA IFEval

Metrics: accuracy Pass@K Reward@1

Baselines: Zero RL (GRPO without guidance) SFT + RL WebSailor-7B ARPO-8B WebThinker-32B-RL MiniMax-M2.1 DeepSeek-V3.2 GPT-5 High OPSD

Builds on: GRPO LUFFY Guide On-policy self distillation ASearcher

Closest papers by meaning

• Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.91)
• Controlling Large Language Model with Latent Action (ICML 2025 · similarity 0.90)
• LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.90)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.90)
• Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization (ICML 2026 · similarity 0.89)
• AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs (NeurIPS 2026 · similarity 0.89)