Research map / Multi-agent systems and games / LLM agents
On the Effects of Data Scale on UI Control Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AndroidControl, a 15k-demonstration Android UI control dataset with high- and low-level instructions, and studies how LoRA fine-tuning of PaLM 2S scales in and out of domain versus zero- and few-shot LLM baselines.
Problem
Whether fine-tuning on human UI demonstrations alone can yield robust Android UI control agents, and how performance scales with data both in-domain and out-of-domain across high- and low-level task instructions.
Approach
A large, diverse Android dataset with both human-written high- and low-level instructions per task, used to empirically fit log-linear scaling trends for LoRA fine-tuning and extrapolate data needs in- and out-of-domain.
Key results
- LoRA-tuned PaLM 2S with all training data reaches 71.5% step accuracy on in-domain high-level instructions (rank 64) and 86.6% on low-level instructions, outperforming zero- and few-shot baselines.
- In-domain extrapolation suggests about 500K and 1M episodes for 95% step accuracy on low- and high-level instructions; out-of-domain projections rise to 10M and 60M episodes.
- Out-of-domain gap grows with data: LT-all-r64 drops 11.2-13.4 pp on high-level OOD splits versus IDD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase LoRA fine-tuning data from 5 to 10k episodes | PaLM 2S UI agent · AndroidControl IDD Random-500, high-level instructions | step accuracy: 30.3 -> 62.0 | Helped |
| Increase LoRA fine-tuning data from 5 to 10k episodes | PaLM 2S UI agent · AndroidControl IDD Random-500, low-level instructions | step accuracy: 57.1 -> 85.7 | Helped |
| Use LoRA rank 64 instead of rank 4 (ablation) | PaLM 2S fine-tuned on all episodes · AndroidControl IDD Random-500, high-level instructions | step accuracy: 65.6 -> 71.5 | Helped |
| Use LoRA rank 64 instead of rank 4 (ablation) | PaLM 2S fine-tuned on all episodes · AndroidControl IDD Random-500, low-level instructions | step accuracy: 81.8 -> 86.6 | Helped |
| Use LoRA rank 64 instead of rank 4 at 10k episodes (ablation) | PaLM 2S UI agent · AndroidControl IDD Random-500, low-level instructions | step accuracy: 85.7 -> 81.9 | Hurt |
| Increase fine-tuning data to 10k episodes | PaLM 2S UI agent · AndroidControl category-unseen, high-level instructions | step accuracy: 25.1 -> 54.2 | Helped |
| Increase fine-tuning data to 10k episodes | PaLM 2S UI agent · AndroidControl category-unseen, low-level instructions | step accuracy: 56.4 -> 75.5 | Helped |
| Train on a mixture of high- and low-level instructions instead of low-level instructions only (ablation) | LoRA-tuned PaLM 2S · AndroidControl, tested on low-level instructions | step-wise accuracy: 83.4 -> 85.4 | Helped |
| Train on a mixture of high- and low-level instructions instead of high-level instructions only (ablation) | LoRA-tuned PaLM 2S · AndroidControl, tested on high-level instructions | step-wise accuracy: 63.6 -> 63.6 | No clear effect |
| Evaluate longer high-level tasks (5 steps to 6 steps) | PaLM-2S-FT · AndroidControl full test split, high-level instructions | episode accuracy: 21.3% to 7.6% | Hurt |
| Fine-tune instead of use zero-shot ER prompting | PaLM 2S vs. PaLM 2L · AndroidControl IDD Random-500, high-level instructions | step accuracy: 33.0 -> 71.5 | Helped |
| Fine-tune instead of use zero-shot ER prompting | PaLM 2S vs. PaLM 2L · AndroidControl IDD Random-500, low-level instructions | step accuracy: 45.9 -> 86.6 | Helped |
Surprising findings
- Out-of-domain performance on high-level tasks degrades relative to in-domain as training data grows, with the IDD-OOD gap widening for larger fine-tuning sets.
- Few-shot prompting with Gemini 1.5 Pro often performs worse than zero-shot methods.
Limitations
- Only one model (PaLM 2S) was fine-tuned, so absolute values may differ across model families.
- Offline step-wise evaluation does not reward alternative routes or corrective actions.
- App category coverage is incomplete relative to all user tasks.
Experimental setup
Datasets: AndroidControl
Benchmarks: AndroidControl
Metrics: step-wise accuracy episode accuracy
Baselines: AitW prompt SeeAct M3A ER prompt zero-shot few-shot Gemini 1.5 Pro
Builds on: SeeAct M3A AitW prompt SeqIO LoRA
Closest papers by meaning
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (ICLR 2025 · similarity 0.88)
- SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.87)
- FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents (ICLR 2026 · similarity 0.87)
- D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies (AAAI 2026 · similarity 0.86)
- OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models (CVPR 2026 · similarity 0.86)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.