Research map / Large language models / Agents, code and math
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WebAgent, a modular LLM agent that pairs a newly pre-trained HTML-specialized encoder-decoder (HTML-T5, local-global attention with long-span denoising) for planning and HTML summarization with Flan-U-PaLM for Python program synthesis, trained via self-experience supervision.
Problem
Web automation on real-world websites is hard due to open-ended action spaces, very long HTML documents exceeding LLM context lengths, and lack of HTML-specific inductive bias in general LLMs.
Approach
Decompose web navigation into closed-loop planning and HTML summarization by a finetuned domain-expert encoder-decoder (HTML-T5 with local/global attention and mixed long-span denoising pretraining), and act via grounded Python/Selenium code generated by a generalist LLM prompted with sub-instructions and HTML snippets.
Key results
- WebAgent reaches 65%, 70%, and 80% success on real-estate, social-media, and map websites, versus 10-20% for single Flan-U-PaLM (over 50% improvement).
- HTML-T5-XL reaches 67.1% on MiniWoB++ with 12K demos, 18.7% above WebN-T5-XL, and 85.6% with 347K demos.
- HTML-T5-XL achieves SoTA on Mind2Web cross-task/website/domain, improving element accuracy by 5-8 points over MindAct (Flan-T5-XL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add both planning and summarization language-model modules (ablation) | Flan-U-PaLM · real-world websites | success: 10.0 -> 65.0; 20.0 -> 70.0; 10.0 -> 80.0 | Helped |
| Add a planning language model (ablation) | Flan-U-PaLM · real-estate | success: 10.0 -> 50.0 | Helped |
| Add a planning language model (ablation) | Flan-U-PaLM · social-media and map | success: 20.0 -> 20.0; 10.0 -> 30.0 | Mixed |
| Add a summarization language model (ablation) | Flan-U-PaLM · real-world websites | success: 10.0 -> 0.0; 20.0 -> 25.0; 10.0 -> 15.0 | Mixed |
| Use local and global attention instead of dense attention (ablation) | Long-T5-Base · MiniWoB++ (L=4096) | success: 35.3% -> 53.6% | Helped |
| Use local and global attention instead of local attention only (ablation) | Long-T5-Base · MiniWoB++ (L=4096) | success: 44.0% -> 53.6% | Helped |
| Use HTML-denoising pre-training (ablation) | LongT5 · MiniWoB++ | success: 53.8% -> 57.0% | Helped |
| Use denoising span lengths 8,64 instead of 3,8,64,Prefix (ablation) | LongT5-Base · real-estate | accuracy: 80.56 -> 82.46 | Helped |
| Use PEGASUS pre-training initialization (ablation) | HTML-T5-Base with extracted HTML · real-estate | accuracy: 67.11 -> 82.46 | Helped |
| Use PEGASUS pre-training initialization (ablation) | HTML-T5-Base with extracted HTML · MiniWoB++ | success: 49.1% -> 57.0% | Helped |
| Extract HTML subtrees around label elements instead of using raw HTML (ablation) | HTML-T5-Base initialized with PEGASUS · MiniWoB++ | success: 56.7% -> 57.0% | No clear effect |
| Apply HTML-denoising pre-training (ablation) | LongT5 · MiniWoB++ | success: LongT5-XL 60.4% -> HTML-T5-XL 67.1% | Helped |
Surprising findings
- Local attention alone beats dense attention by about 9% on MiniWoB++, suggesting local element-attribute relations matter for web tasks.
- Flan-LongT5 does not consistently outperform Flan-T5 on reasoning tasks, and instruction tuning gains on HTML tasks are small compared with HTML-denoising.
Limitations
- Planning errors dominate failures on long-horizon tasks, e.g. 70% of real-estate failures.
- Flan-U-PaLM (540B) is too large to incorporate execution feedback for program synthesis.
- Real-world evaluation is costly and relies on human supervision; evaluated on only three anonymized websites.
Experimental setup
Datasets: CommonCrawl HTML (100 WARC files, April 2019) MiniWoB++ 12K demonstrations MiniWoB++ 347K demonstrations Mind2Web WebSRC C4
Benchmarks: real-world websites (real-estate, social-media, map) MiniWoB++ Mind2Web WebSRC description generation benchmark
Metrics: success rate score element accuracy operation F1 step success rate EM F1 accuracy BLEU ROUGE-1
Baselines: Flan-U-PaLM Flan-U-PaLM+P Flan-U-PaLM+S WebN-T5-XL Flan-T5-XL MindAct Synapse MarkupLM-Large TIE-Large
Builds on: LongT5 Flan-U-PaLM MindAct PEGASUS
Closest papers by meaning
- How to Train Your LLM Web Agent: A Statistical Diagnosis (NeurIPS 2025 · similarity 0.90)
- Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents (ICLR 2025 · similarity 0.90)
- WALT: Web Agents that Learn Tools (ICLR 2026 · similarity 0.89)
- WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents (ICLR 2026 · similarity 0.89)
- RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users (AAAI 2026 · similarity 0.89)
- WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.