Research map / Multi-agent systems and games / LLM agents
FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FingerTip 20K, a benchmark of 20K real-life Android demonstrations from 95 users with contextual and historical data, evaluating proactive task suggestion and personalized task execution, and shows LoRA fine-tuning of Qwen-2.5-VL-7B improves both tracks.
Problem
Mobile GUI agents are passive, relying on explicit instructions, and ignore user context, history, and action preferences; existing benchmarks lack user-related contextual and longitudinal data.
Approach
Collect longitudinal, in-the-wild user demonstrations on physical Android phones with intents, time, location, and user profiles, and define two tracks: proactive intent prediction from context/history and personalized execution that mimics user action preferences measured against other users' action sequences.
Key results
- Best generalist model Qwen-QVQ-Max reaches 12.8% SR1 on proactive suggestion vs 30.3% for humans.
- UI-TARS-1.5-7B achieves 38.5% SR2 on personalized execution, the highest among evaluated models.
- LoRA fine-tuning Qwen-2.5-VL-7B on the full training set raises SR1 from 3.1% to 26.0% and Sim2 from 0.95 to 1.42.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| LoRA fine-tuning of Qwen-2.5-VL-7B on the collected data (full training set, rank 64) | Qwen-2.5-VL-7B · Proactive task suggestion | SR1 (%): +22.9 (3.1 to 26.0) | Helped |
| LoRA fine-tuning of Qwen-2.5-VL-7B on the collected data (full training set, rank 64) | Qwen-2.5-VL-7B · Personalized task execution | Sim2: +0.47 (0.95 to 1.42) | Helped |
| LoRA fine-tuning of Qwen-2.5-VL-7B on the collected data (full training set, rank 64) | Qwen-2.5-VL-7B · Personalized task execution | SR2 (%): +14.0 (1.5 to 15.5) | Helped |
| Increasing training data from 1k to full 16k episodes in fine-tuning (ablation) | Qwen-2.5-VL-7B LoRA · Proactive task suggestion | SR1 (%): 9.7 (r4, 1k) to 20.3 (r4, all) | Helped |
| Increasing LoRA rank from 4 to 64 (ablation) | Qwen-2.5-VL-7B fine-tuning on 1k episodes · Proactive task suggestion | SR1 (%): 9.7 to 11.8 | Helped |
| Fine-tuned model is weaker than UI-TARS-1.5-7B on execution success | Qwen-2.5-VL-7B fine-tuning vs UI-TARS · Personalized task execution | SR2 (%): 15.5 vs 38.5 | Hurt |
| Training on one track (proactive or personalized) and testing on the other track (ablation) | Fine-tuned Qwen-2.5-VL-7B · Cross-track evaluation | SR1 (%) and SR2 (%): SR2 1.0 on proactive-trained model vs 12.5 on personalized-trained | Hurt |
| Joint fine-tuning on both tracks instead of separate per-track fine-tuning (ablation) | Qwen-2.5-VL-7B · Both tracks | SR1 (%) / SR2 (%): SR1 9.2 vs 9.7; SR2 11.0 vs 12.5 | Hurt |
| Adding the first 3 screenshots of the current action to input (vs zero screenshots) for GPT-4.1 (ablation) | GPT-4.1 proactive task suggestion · Proactive task suggestion | SR1 (%): 7.2 to 9.9 (with 20 history intents) | Helped |
| Removing historical intents (cold-start) while keeping 3 screenshots (ablation) | GPT-4.1 proactive task suggestion · Proactive task suggestion | SR1 (%): 4.3 vs 9.9 with history | Hurt |
| Removing time from the input context (ablation) | GPT-4.1 proactive task suggestion · Proactive task suggestion | SR1 (%): 4.1 vs 7.2 | Hurt |
| Removing scenario from the input context (ablation) | GPT-4.1 proactive task suggestion · Proactive task suggestion | SR1 (%): 6.1 vs 7.2 | Hurt |
Surprising findings
- Adding screenshots did not always help: 3 screenshots with no history gave only 4.3% SR1, far below 7.2% with 20 historical intents.
- Fine-tuned Qwen-2.5-VL-7B has lower SR2 than UI-TARS-1.5-7B, though it reaches higher personalization similarity.
Limitations
- All 95 contributors are in mainland China and mostly use Chinese apps, limiting generalizability.
- Fine-tuning only tested on a single 7B model due to cost.
- Privacy risk: fine-grained UI traces may allow re-identification even after anonymization.
Experimental setup
Datasets: FingerTip 20K
Benchmarks: FingerTip 20K
Metrics: SR1 Sim1 SR2 Sim2 step ratio time token count
Baselines: GPT-4.1 Qwen-VL-Max DeepSeek-VL2 Qwen-2.5-VL-7B Qwen-2.5-VL-72B Qwen-QVQ-Max AutoDroid AppAgent Aguvis-7B CogAgent-9B UI-TARS-1.5-7B
Builds on: AppAgent AutoDroid
Closest papers by meaning
- SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.91)
- ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices (CVPR 2026 · similarity 0.90)
- Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · similarity 0.90)
- VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.89)
- CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs (NeurIPS 2025 · similarity 0.89)
- ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.