Research map / Video / Video-language understanding
Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MONDAY, a 313K-frame cross-platform mobile OS navigation dataset auto-extracted from 20K YouTube instructional videos using OCR scene detection and GPT-4o-based multi-step action identification.
Problem
Mobile OS GUI agents lack diverse, real-world, cross-platform training data; manual annotation is costly and static datasets become obsolete as OS versions change.
Approach
Automatically mine navigation procedures from instructional YouTube videos: OCR-based text-change scene transition detection, GroundingDINO+PaddleOCR UI element detection with Set-of-Marks, and a 3-step GPT-4o action identification (scene summary, initial action, zoomed refinement) guided by narrations.
Key results
- OCR-based scene transition detection achieves 95.04% F1 vs 82.27% (SceneDetect) and 70.86% (YUV-diff).
- UI element detection hit ratio of 99.87% vs 91.83% for OmniParser.
- MONDAY-induced SeeClick and Llama-3.2 variants outperform originals on the unseen Windows Mobile test set, e.g., SeeClick AitW-finetuned 35.61 to 47.56 and Llama-3.2 AMEX-finetuned 23.17 to 47.32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain SeeClick on MONDAY | SeeClick, then finetune on AitW · Windows Mobile | accuracy: 35.61 -> 47.56 | Helped |
| Pretrain SeeClick on MONDAY | SeeClick, then finetune on AMEX · AMEX | accuracy: 68.19 -> 80.00 | Helped |
| Pretrain SeeClick on MONDAY | SeeClick, then finetune on AitW · MONDAY | accuracy: 37.45 -> 60.71 | Helped |
| Use OCR-based text change detection instead of visual-feature transition detection | Scene transition detection · Scene transition detection evaluation dataset | F1-score: 82.27 -> 95.04 | Helped |
| Use mobile-specific filtering with GroundingDINO and OCR | UI element detection · UI element detection evaluation dataset | Hit Ratio: 91.83 -> 99.87 | Helped |
| Add the final refinement step (ablation) | Multi-image 3-step action identification · Action identification evaluation dataset | accuracy (Touch): 89.97 -> 91.84 | Helped |
| Use multi-stage reasoning instead of direct one-step action identification (ablation) | Action identification · Action identification evaluation dataset | accuracy (Touch): 74.67 -> 91.84 | Helped |
| Add narration (ablation) | Multi-image 3-step action identification · Action identification evaluation dataset | accuracy (All): 78.20 -> 80.90 | Helped |
| Use temporal context from multiple images rather than a single image (ablation) | Action identification · Action identification evaluation dataset | accuracy (All): 77.22 -> 80.90 | Helped |
Surprising findings
- Video-derived MONDAY pretraining improves models even on AitW and AMEX test sets, not just the new platform.
Limitations
- Action identification relies on proprietary GPT-4o.
- Windows Mobile evaluation set is small (50 videos, 554 actions).
- Dataset is restricted to iOS and Android phones with English narration.
Experimental setup
Datasets: MONDAY AitW AMEX C4 Dolma RICO
Benchmarks: AitW AMEX MONDAY Windows Mobile (unseen platform)
Metrics: scene transition F1-score hit ratio action matching accuracy touch action accuracy
Baselines: SceneDetect YUV-diff OmniParser Video2Action SeeClick Llama-3.2-11B-Vision-Instruct
Builds on: Set-of-Marks Android in the Wild SeeClick GroundingDINO Video2Action
Closest papers by meaning
- Watch and Learn: Learning to Use Computers from Online Videos (CVPR 2026 · similarity 0.86)
- Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining (ICML 2026 · similarity 0.84)
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.84)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.84)
- Generative Visual Code Mobile World Models (ICML 2026 · similarity 0.84)
- Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation (ICML 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.