Research map / Vision-language and multimodal / Robot action and navigation
Lightweight Neural App Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LiMAC, a lightweight app-control architecture pairing a ~520M GPT-2-style Action Transformer (AcT) that predicts action types and click targets with a fine-tuned VLM (Florence2 or Qwen2-VL) for text generation.
Problem
Smartphone app agents built on large foundation models like GPT-4o are slow and expensive, and impractical to run on resource-constrained phones for everyday mobile control.
Approach
A gated pipeline where a small transformer over embedded goal, UI elements (fine-tuned CLIP image + text + attribute embeddings) and action tokens predicts the action type and, for clicks, the target element via an InfoNCE contrastive objective; a fine-tuned VLM is invoked only for input-text and open-app actions.
Key results
- LiMAC with Florence2 reaches 63.1 overall on AndroidControl and 72.2 on AitW, versus 57.0 and 70.8 for Florence2 alone.
- LiMAC with Qwen2-VL improves AitW overall from 51.0 to 70.9 and AndroidControl from 52.2 to 62.5, at 0.63 s average inference time versus 3.03 s for Qwen2-VL alone.
- GPT-4o prompt baselines (T3A, M3A, SeeAct) achieve 26.9-42.5 on AitW and 29.9-57.5 on AndroidControl overall, below the LiMAC variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AcT to Florence2 for action-type and click-target prediction | Florence2 · AitW | overall accuracy: 70.8 -> 72.2 | Helped |
| add AcT to Florence2 for action-type and click-target prediction | Florence2 · AndroidControl | overall accuracy: 57.0 -> 63.1 | Helped |
| add AcT to Qwen2-VL for action-type and click-target prediction | Qwen2-VL · AitW | overall accuracy: 51.0 -> 70.9 | Helped |
| add AcT to Qwen2-VL for action-type and click-target prediction | Qwen2-VL · AndroidControl | overall accuracy: 52.2 -> 62.5 | Helped |
| add AcT for action-type prediction and click-target prediction instead of T3A | T3A · AitW | overall accuracy: 26.9 -> 69.8 | Helped |
| add AcT for action-type prediction and click-target prediction instead of M3A | M3A · AitW | overall accuracy: 35.6 -> 70.0 | Helped |
| use M3A instead of AcT for click-target prediction | LiMAC with T3A text generation · AndroidControl | overall accuracy: 63.2 -> 67.4 | Helped |
| use original CLIP instead of fine-tuned CLIP (ablation) | LiMAC with Florence2 · AndroidControl | overall accuracy: 63.1 -> 60.0 | Hurt |
| add CLIP fine-tuning (ablation) | LiMAC with Florence2 · AndroidControl | overall accuracy: 60.0 -> 63.1 | Helped |
| remove image embeddings (ablation) | LiMAC with Florence2 · AndroidControl | overall accuracy: 63.1 -> 56.0 | Hurt |
| remove UI text embeddings (ablation) | LiMAC with Florence2 · AndroidControl | overall accuracy: 63.1 -> 63.0 | No clear effect |
Surprising findings
- Removing the UI text embeddings barely hurts AcT (overall 63.0 vs 63.1 on AndroidControl), suggesting screenshots alone suffice.
- Using M3A (GPT-4o) for click targeting raises AndroidControl overall to 66.8-67.4 but adds large latency, showing the best accuracy trades off against speed.
Limitations
- Training data is limited (13K and 18K episodes) and there is no pretraining.
- Only click and long-press targets are handled by AcT-style contrastive targeting; text actions depend on the VLM.
- No online interaction or reinforcement learning; error recovery and safety are not addressed.
Experimental setup
Datasets: AndroidControl Android-in-the-Wild (AitW)
Benchmarks: AndroidControl Android-in-the-Wild (AitW)
Metrics: relaxed overall action accuracy action-type accuracy click-target accuracy text accuracy average inference time
Baselines: T3A M3A SeeAct Florence2 Qwen2-VL
Builds on: CLIP InfoNCE ReAct SeeAct GPT-2
Closest papers by meaning
- SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.83)
- On the Effects of Data Scale on UI Control Agents (NeurIPS 2024 · similarity 0.83)
- Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control (NeurIPS 2025 · similarity 0.83)
- DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agent (ICLR 2025 · similarity 0.82)
- MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulation (CVPR 2025 · similarity 0.81)
- FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents (ICLR 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.