Research map / Multi-agent systems and games / LLM agents

OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OS-Oracle, a cross-platform GUI critic framework with a 310k-sample synthetic negative-mining data pipeline, SFT plus CP-GRPO training on Qwen2.5-VL-7B, and the 738-instance human-labeled OS-Critic Bench.

Problem

Computer-using agents make step-level errors in long-horizon GUI tasks, but critic models for them lack diverse, high-quality cross-platform training data and public step-level critic benchmarks.

Approach

Synthesize type-specific hard negatives (operation failure, inefficient error recovery, mistimed termination, inaccurate localization) from positive trajectory steps, train a pointwise critic with SFT then CP-GRPO whose consistency reward ties rationale polarity to the final Yes/No judgment.

Key results

• OS-Oracle-7B reaches 68.02% overall accuracy and 72.81 F1 on OS-Critic Bench, the best among open-source models, and 70.78% accuracy on Mobile, above the proprietary models listed.
• With UI-TARS-1.5-7B as the agent, OS-Oracle-7B as pre-critic raises OSWorld success from 29.2% to 31.0%.
• CP-GRPO raises reasoning-judgment consistency from 90.51% (SFT+GRPO) to 99.59% at comparable accuracy (65.31 vs 66.12).

Surprising findings

• GPT-4o as a pre-critic degraded CUA performance in dynamic evaluation, while OS-Oracle-7B improved it.
• GPT-annotated negatives hurt over the base Qwen2.5-VL-7B (55.42 vs 58.27 accuracy), while synthetic negatives improved it to 60.03.

Limitations

• OS-Critic Bench is small (738 instances) and its actions are sampled from a single model, Qwen2.5-VL-7B.
• Consistency reward relies on a rule-based lexicon with a Qwen3-8B fallback, which may be imprecise.
• Dynamic evaluation is limited to one agent (UI-TARS-1.5-7B) and a three-retry regeneration scheme.

Experimental setup

Datasets: AndroidControl GUI-Odyssey AMEX AITZ Mind2Web ScaleCUA-Web AgentNet AndroidControl test GUI-Odyssey test guiact test AgentNet-Bench

Benchmarks: OS-Critic Bench AndroidWorld OSWorld

Metrics: accuracy F1-score consistency success rate

Baselines: GPT-4o GPT-5 Claude-4.5-Sonnet Gemini-2.5-Pro Qwen2.5-VL-7B Qwen3-VL-8B UI-TARS-1.5-7B ScaleCUA-7B ScaleCUA-32B SE-WSM-7B GUI-Critic-R1

Builds on: GRPO Qwen2.5-VL OmniParser V2

Closest papers by meaning

• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.90)
• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation (NeurIPS 2025 · similarity 0.89)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.88)
• GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.88)