Research map / Large language models / Agents, code and math
CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CORE, a cloud-local collaborative framework for mobile agents that uses XML layout-aware block partitioning, local-LLM block ranking and sub-task candidates, and multi-round accumulation so the cloud LLM sees only a subset of UI blocks, cutting UI exposure by up to 55.6% with success near GPT-4o.
Problem
Cloud-based LLM mobile agents upload the full UI state at every step, exposing redundant and sensitive information, while local LLMs that avoid uploads have much lower task success rates.
Approach
An asymmetric collaboration in which a local LLM with full UI access partitions and ranks XML-derived UI blocks and proposes sub-task candidates, while a cloud LLM without direct UI access plans and selects elements, requesting more blocks incrementally only when needed.
Key results
- On DroidTask, GPT-4o + Gemma2-9B achieves 69.23% success (vs 74.13% for GPT-4o) with 55.60% UI exposure reduction.
- On AndroidLab, GPT-4o + Qwen2.5-7B achieves 41.84% success (vs 44.90% for GPT-4o) with 29.97% reduction.
- Sensitive UI elements uploaded to the cloud drop by 70.49% on DroidTask and 38.84% on AndroidLab.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CORE with GPT-4o + Gemma2-9B instead of GPT-4o alone | Mobile agent · DroidTask | Reduction Rate: 0.00% -> 55.60% | Helped |
| Use CORE with GPT-4o + Gemma2-9B instead of GPT-4o alone | Mobile agent · DroidTask | Success Rate: 74.13% -> 69.23% | Hurt |
| Use CORE with GPT-4o + Qwen2.5-7B instead of GPT-4o alone | Mobile agent · AndroidLab | Reduction Rate: 0.00% -> 29.97% | Helped |
| Use CORE with GPT-4o + Qwen2.5-7B instead of GPT-4o alone | Mobile agent · AndroidLab | Success Rate: 44.90% -> 41.84% | Hurt |
| Add XML-layout-aware block partitioning (ablation) | CORE (GPT-4o + Gemma2-9B) · DroidTask | Success Rate: 62.24% -> 69.23% | Helped |
| Add XML-layout-aware block partitioning (ablation) | CORE (GPT-4o + Gemma2-9B) · DroidTask | Reduction Rate: 53.44% -> 55.60% | Helped |
| Add collaborative co-planning (ablation) | CORE (GPT-4o + Gemma2-9B) · DroidTask | Success Rate: 59.44% -> 69.23% | Helped |
| Add collaborative co-planning (ablation) | CORE (GPT-4o + Gemma2-9B) · DroidTask | Reduction Rate: 48.67% -> 55.60% | Helped |
| Add multi-round interactions (ablation) | CORE (GPT-4o + Gemma2-9B) · DroidTask | Success Rate: 36.36% -> 69.23% | Helped |
| Add the accumulation mechanism (ablation) | CORE (GPT-4o + Gemma2-9B) · DroidTask | Success Rate: 53.15% -> 69.23% | Helped |
| Use local LLM ranking instead of random block ordering (ablation) | CORE (GPT-4o + Gemma2-9B) · DroidTask | Success Rate: 51.75% -> 69.23% | Helped |
| Use local LLM ranking instead of basic-order block ordering (ablation) | CORE (GPT-4o + Gemma2-9B) · DroidTask | Reduction Rate: 32.59% -> 55.60% | Helped |
Surprising findings
- CORE surpasses the GPT-4o-only baseline on some apps (e.g., Firefox, Map), attributed to filtering out distracting UI elements.
- Overall latency is 1.49-1.66x the GPT-4o baseline, and cloud token usage is 1.10-1.15x on DroidTask, despite the reduced upload.
Limitations
- Overall latency and cloud token usage increase relative to cloud-only agents.
- Local models are weak; performance depends on local LLM ranking quality.
- XML-only pipeline; does not handle screenshot-based or vision-only settings.
Experimental setup
Datasets: DroidTask AndroidLab
Benchmarks: DroidTask AndroidLab
Metrics: task success rate UI reduction rate sensitive element reduction latency cloud token usage
Baselines: GPT-4o cloud-only Gemma2-9B local-only Qwen2.5-7B local-only LLaMA3.1-8B local-only Basic-order ranking Random ranking
Builds on: AutoDroid
Closest papers by meaning
- FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents (ICLR 2026 · similarity 0.89)
- SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.87)
- VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.87)
- AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (ICML 2026 · similarity 0.87)
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (ICML 2024 · similarity 0.86)
- Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.