Research map / Vision-language and multimodal / Robot action and navigation
D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI
Suhwan Choi, Jaeyoon Jung, Haebin Seong, Minchan Kim, Minyeong Kim, Yongjun Cho, Yoonshik Kim, Park Yu Been, Youngjae Yu, Yunsung Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add desktop vision-action pretraining (VAPT, human data only) | InternVL3-1B · LIBERO (Total) | success rate (%): 84.8 -> 96.6 | Improved |
| add desktop vision-action pretraining (VAPT, human data only) | InternVL3-1B · LIBERO (10 long) | success rate (%): 54.2 -> 93.6 | Improved |
| add 1K+ hours of pseudo-labeled YouTube gameplay to pretraining (ablation) | VAPT · LIBERO (Total) | success rate (%): 96.6 -> 92.2 | Hurt |
| add desktop pretraining | InternVL3-1B · Meta-World (Avg) | success rate (%): 19.9 -> 25.0 | Improved |
| add pseudo-labeled data to pretraining (ablation) | VAPT · Meta-World (Avg) | success rate (%): 25.0 -> 24.8 | No clear effect |
| add desktop pretraining | InternVL3-1B · SO101 real-world pick-and-place | success rate (%): 70.0 -> 80.0 | Improved |
| add pseudo-labeled data to pretraining (ablation) | VAPT · CANVAS navigation (Total) | success rate (%): 75.3 -> 83.3 | Improved |
| add human-only desktop pretraining | InternVL3-1B · CANVAS navigation (Total) | success rate (%): 75.3 -> 75.3 | No clear effect |
| use Generalist-IDM instead of per-game Specialist IDM | IDM · in-distribution games, keyboard accuracy | accuracy (%): 28.80 -> 86.36 (Brotato) | Improved |
| use temporal offset tau=0 vs future context (NEP-tau) (ablation) | Generalist-IDM · 6 in-distribution games | Pearson X: 0.108 -> 0.803 (tau 0 -> 50 ms) | Improved |
| increase tau from 100 to 200 ms (ablation) | Generalist-IDM · 6 in-distribution games | Pearson X / keypress: 0.796 -> 0.893 Pearson X; 0.730 -> 0.740 Kbd | No clear effect |
| use H.265 video storage instead of raw frames | OWAMcap · Minecraft screen capture | compression ratio: 1.0x -> 217.8x | Improved |
Closest papers by meaning
- Toward Embodied World Agents via Embodied-Planning Dataset and Interactive World Models (NeurIPS 2026 · similarity 0.89)
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter (NeurIPS 2026 · similarity 0.88)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.88)
- OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds (ICLR 2026 · similarity 0.88)
- Universal Actions for Enhanced Embodied Foundation Models (CVPR 2025 · similarity 0.87)
- VirtualEnv: A Platform for Embodied AI Research (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
