Research map / Multi-agent systems and games / LLM agents

OpenApps: Simulating Environment Variations to Measure UI Agent Reliability

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OpenApps, a lightweight Python/FastHTML simulator of six configurable apps, and uses it to show that UI-agent success and behaviors vary strongly across app appearance and content variations, which fixed-clone benchmarks miss.

Problem

Existing UI-agent benchmarks measure reliability only within fixed app clones, so they cannot capture how agent success and behavior change across app variations (appearance, content, resolution) that agents encounter when deployed.

Approach

A light-weight, fully observable, Python-native app ecosystem whose appearance and content are configured via YAML, enabling thousands of app variations with state-based rewards and a reliability metric that measures deviation across variations rather than within a fixed app version.

Key results

• Kimi-VL-A3B-Instruct average success fluctuates from 63% to 4% across app variations.
• UI-TARS is about 5x more likely to hallucinate actions depending on app variation.
• Qwen2.5-VL intent misunderstanding rises from 3% in default to 40-45% with long or adversarial content.
• Over 10,000 trials across seven agents were run.

Surprising findings

• Within-fixed-app reliability underestimates fluctuations; for Qwen2.5-VL, Kimi-VL, and UI-TARS the cross-variation std is more than twice the within-app std.
• Higher screen resolution does not always help; under dark theme, HD yields a large drop in UI-TARS success versus FHD.

Limitations

• Only 15 simple tasks requiring few steps, not representative of real-world complexity.
• Variations are studied one factor at a time; interactions only explored in a small experiment.
• Limited set of agents and small per-task trial counts (three seeds).

Experimental setup

Datasets: OpenApps app configurations (YAML) and 15 tasks

Benchmarks: OpenApps

Metrics: pass@1 standard deviation of rewards mean absolute deviation loop count hallucinated action rate intent misunderstanding rate

Baselines: Claude 4 Sonnet GPT-4o Qwen2.5-VL Kimi-VL-A3B-Instruct UI-TARS-1.5-7B LLaVA-v1.6-mistral-7b

Builds on: BrowserGym FastHTML

Closest papers by meaning

• AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (ICML 2026 · similarity 0.89)
• Computer Use at the Edge of the Statistical Precipice (NeurIPS 2026 · similarity 0.88)
• VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.88)
• Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · similarity 0.87)
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (ICLR 2025 · similarity 0.87)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.87)