Research map / Multi-agent systems and games / LLM agents

ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ProactiveMobile, a 3,660-instance benchmark for proactive mobile agents that maps multi-dimensional on-device context to executable function sequences over a 63-API pool, and fine-tunes Qwen2.5-VL-7B to 20.82% success rate, above o1 (17.02%).

Problem

Proactive mobile agents, which anticipate user needs and act without explicit commands, lack benchmarks that reflect real-world complexity and allow objective, executable evaluation. Existing benchmarks assume a single correct action and rely on natural-language outputs that cannot be executed.

Approach

Formalize proactive intelligence as inferring latent intents from four context dimensions (user profile, device status, world information, behavioral trajectories) and requiring an executable function-call sequence from a 63-API pool. Each instance has one to three valid target actions, and success is scored by functional equivalence with a best-match protocol, alongside a false trigger rate for n

Key results

• Fine-tuned Qwen2.5-VL-7B-Instruct reaches 20.82% success rate on ProactiveMobile test, above o1 (17.02%), GPT-5 (11.37%), and Gemini-2.5-Pro (9.62%).
• On the out-of-distribution set (64 instances from Logistics Delivery and Smart Home), fine-tuned Qwen reaches 20.31% SR versus o1 at 15.63%.
• Fine-tuning lifts MiMo-VL-7B-SFT-2508 from 1.31% to 13.47% SR; removing the recommendation text from the output drops Qwen SR to 9.18% with FTR near 93%.

Surprising findings

• Training the model to output only functions, or to add a Think step before functions, yields false trigger rates near 93–95% in the function-only variants, showing the textual recommendation is needed to learn when not to act.
• A small fine-tuned 7B model outperforms o1 and GPT-5 on the proactive task, despite the absolute success rates remaining around 21%.

Limitations

• Absolute success rates remain low (about 21% best), so the trained models are not yet reliable enough for on-device deployment.
• Multimodal (screenshot-based) tasks lag text tasks substantially (15.61% vs 26.04% SR for the best model).
• Evaluation depends on an LLM judge (Gemini-2.5-Pro) for functional equivalence, with reported 98% agreement with human experts on a sample.

Experimental setup

Datasets: GUI-Odyssey AITZ CAGUI MobileAgentBench

Benchmarks: ProactiveMobile

Metrics: success rate false trigger rate F1

Baselines: GPT-5 GPT-4o o1 Gemini-2.5-Pro Qwen2.5-VL-7B-Instruct MiMo-VL-7B-SFT-2508

Builds on: ProactiveAgent FingerTip-20K LLaVA

Closest papers by meaning

• FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents (ICLR 2026 · similarity 0.90)
• SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.90)
• ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response (ICML 2026 · similarity 0.90)
• Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance (ICLR 2025 · similarity 0.89)
• ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions (NeurIPS 2025 · similarity 0.88)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.88)