Research map / Reinforcement learning / Visual and representation RL

StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: In-context and adaptive RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes StraTA, which samples a compact trajectory-level strategy from the initial state, conditions all actions on it, and trains with hierarchical GRPO-style rollouts plus diverse strategy sampling and self-judgment rewards, reaching SOTA on ALFWorld, WebShop, and SciWorld.

Problem

Long-horizon agentic RL for LLMs is hard because purely reactive policies weaken exploration and credit assignment under sparse, delayed rewards.

Approach

Generate one natural-language strategy at episode start, prepend it to every step's prompt, and optimize strategy and action levels jointly with two-level GRPO groups (strategy reward = mean of top-δ rollout outcomes), plus farthest-point-sampled diverse strategies and a step-level penalty from the agent's own critical self-judgment.

Key results

• StraTA with Qwen2.5-7B reaches 93.1% success on ALFWorld and 84.2% success on WebShop, versus 90.8% (GiGPO) on ALFWorld.
• With Qwen2.5-1.5B, StraTA reaches 90.7% ALFWorld and 82.5% WebShop success.
• On SciWorld, StraTA attains a 63.5% overall score, above ScalingInter (57.0%) and Claude-4-Sonnet (57.4%), with 100.0% on Lifespan.

Surprising findings

• Frontier closed-source models (GPT-5.1, Claude-4-Sonnet, Gemini-2.5-Flash) score far below RL-trained 1.5B/7B models on ALFWorld and WebShop.
• Ablation: the diverse strategy rollout alone raised 3B ALFWorld success from 79.0% to 87.9%, and increasing oversampling ratio sigma steadily improved results.

Limitations

• Effectiveness depends on the quality of the generated strategy.
• A fixed strategy may become restrictive when the environment changes substantially during an episode.
• Self-judgment has residual misjudgments (accuracy 90.1% at step 50).

Experimental setup

Datasets: ALFWorld WebShop SciWorld

Benchmarks: ALFWorld WebShop SciWorld

Metrics: success rate task score per-subtask success rate sample efficiency

Baselines: ReAct PPO RLOO GRPO GiGPO GPT-5.1 Claude-4-Sonnet Gemini-2.5-Flash AgentGym-RL ScalingInter

Builds on: GRPO GiGPO PPO RLOO DAPO

Closest papers by meaning

• Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.89)
• Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents (ICML 2026 · similarity 0.88)
• Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
• Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.88)
• Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Search (ICLR 2025 · similarity 0.87)
• Strategic Planning: A Top-Down Approach to Option Generation (ICML 2025 · similarity 0.87)