Research map / Multi-agent systems and games / LLM agents

WARC-Bench: Web Archive based Benchmark for GUI Subtask Executions

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces WARC-Bench, a 438-task web-archive-based benchmark for GUI subtasks, and trains Qwen2.5-VL 7B/72B agents with SFT followed by PPO-based RLVR on synthetic tasks.

Problem

Web agents must master short-horizon GUI subtasks (date pickers, scrolling containers, form fields), but existing benchmarks either target long-horizon tasks, use simplified synthetic widgets, or evaluate single-step grounding, leaving realistic subtask execution under-evaluated.

Approach

Replayable Web ARChive (WARC) recordings of real and synthetic websites serve as sandboxed, deterministic environments with programmatic end-state rewards; a screenshot-only Subtask Vision Agent (SVA) is used for evaluation, and Qwen2.5-VL models are improved via SFT on distilled trajectories and online PPO with verifiable rewards.

Key results

• Claude Sonnet 4.0 (SVA) reaches 64.83% on the WARC-Bench test set, the highest among evaluated models.
• Ours-72B-RLVR reaches 52.33% on test, up from 48.33% for Ours-72B-SFT and 37.33% for base Qwen2.5-VL 72B.
• Ours-7B-RLVR improves test accuracy from 4.67% base to 29.17%, with SFT at 27.33%.

Surprising findings

• Dedicated computer-use agents (e.g., Claude Sonnet 4.0 CUA at 47.17%) underperform the simple SVA design with the same model (64.83%) on the test set.
• Offline GRPO and PPO variants underperform online PPO, and in the 7B case offline PPO (61.76%) falls below SFT (66.54% dev).

Limitations

• WARC replay cannot render unrecorded interactions and fails on Cloudflare or robot-blocking sites.
• Test set is small (200 real-website tasks), and OpenCUA results are single-run.
• Human baseline covers only 60 dev tasks with 3 participants.

Experimental setup

Datasets: WARC-Bench train split (1059 tasks) distilled teacher trajectories (~12k)

Benchmarks: WARC-Bench WebArena-Lite MiniWoB++ ScreenSpot V2

Metrics: trajectory-level success rate

Baselines: Claude Sonnet 4.0 computer-use OpenAI CUA GPT-5 UI-TARS 1.5 7B OpenCUA 7B OpenCUA 32B Qwen2.5-VL 7B Qwen2.5-VL 72B

Builds on: BrowserGym PPO Qwen2.5-VL WebArena Mind2Web

Closest papers by meaning

• WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.89)
• RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users (AAAI 2026 · similarity 0.89)
• Multimodal Web Navigation with Instruction-Finetuned Foundation Models (ICLR 2024 · similarity 0.89)
• Go-Browse: Training Web Agents with Structured Exploration (ICLR 2026 · similarity 0.88)
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.88)
• TimeWarp: Evaluating Web Agents by Revisiting the Past (NeurIPS 2026 · similarity 0.88)