Research map / Multi-agent systems and games / LLM agents

A Subgoal-driven RL Framework for Improving Long-Horizon Web Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MiRA, an offline-to-online RL framework with subgoal-based potential shaping and a Gemini-2.5-pro subgoal planner (SGO) for long-horizon web agents, evaluated on WebArena-Lite.

Problem

Long-horizon web navigation agents get stuck midway and suffer from sparse binary rewards that make RL credit assignment difficult.

Approach

Generate teacher-LLM subgoals, use them both for inference-time milestone checking/replanning (SGO) and as progress labels to train a potential critic whose differences give auxiliary shaping rewards in a KL-regularized regression (MSE on log-ratio) policy update, iterated with failure-driven curriculum.

Key results

• Gemma3 + MiRA (12B) reaches 43.0% average success rate on WebArena-Lite, versus 35.1% for Gemma3 + WebRL and 33.3% for Gemma3 + DigiRL.
• Gemini-2.5-pro-SGO improves average success from 23.0% to 32.1% on WebArena-Lite.
• SGO reduces Stuck Midway failures from 48.41% (Gemini 2.5 Pro) to 39.87%.

Surprising findings

• Subgoal-based potential shaping alone is not evaluated as strictly policy-invariant; the paper uses alpha=0.3 chosen by grid search rather than a theoretical guarantee for the learned potential.
• Naively increasing static thinking budget yields diminishing returns, with 16384 tokens dropping success to about 26%.

Limitations

• Proprietary Gemini-2.5-pro teacher and judge are required for subgoal generation and checking, limiting reproducibility and adding cost.
• Evaluation restricted to WebArena-Lite (165 tasks), so variance on small domains is high (e.g., Map has 26 tasks).

Experimental setup

Datasets: WebArena-Lite WebArena WebRL task pool

Benchmarks: WebArena-Lite

Metrics: success rate pass@k failure mode distribution

Baselines: GPT-4-Turbo GPT-4o Gemini-2.5-flash Gemini-2.5-pro AutoWebGLM Llama3.1 + WebRL Gemma3 + DigiRL Gemma3 + WebRL AWR DigiRL

Builds on: WebRL DigiRL Advantage-Weighted Regression Potential-Based Reward Shaping DPO

Closest papers by meaning

• WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents (ICLR 2026 · similarity 0.90)
• Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (ICLR 2025 · similarity 0.90)
• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning (ICLR 2025 · similarity 0.90)
• Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.89)
• WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.89)
• AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents (ICLR 2025 · similarity 0.88)