Research map / Multi-agent systems and games / LLM agents
OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniActor, a generalist GUI and embodied agent built on Qwen2-VL that shares shallow-layer parameters and separates deep-layer experts (Layer-heterogeneity MoE) to exploit synergy and avoid conflict between GUI and robot data.
Problem
Training a single multimodal agent on both 2D GUI and 3D embodied data causes performance degradation due to conflicting action spaces, while the two share useful environment and instruction understanding.
Approach
Layer-heterogeneity MoE: shared attention/FFN in shallow layers (up to layer 8) where GUI and embodied parameter updates are similar, and separate attention/FFN experts plus output heads for deep layers, with a unified vocabulary-based action space (embodied actions discretized into reserved low-frequency tokens).
Key results
- OmniActor reaches 69.5% success on LIBERO-90, 6.1 points above OmniActor-EA (63.4%), and 86.4 / 77.5 / 66.0 on AndroidControl-Low / High and GUI Odyssey.
- Directly mixing GUI and embodied data drops LIBERO-90 to 50.5%, versus 63.4% for embodied-only training, showing the data conflict.
- Switching the base model to Qwen2.5-VL-7B raises the average over four tasks from 74.9 to 78.4, with GUI Odyssey rising to 81.0.
Surprising findings
- Fully separated experts (OmniActor hard) underperform the layer-heterogeneity design, indicating that sharing shallow layers gives a real benefit beyond removing conflict.
- Embodied-only data training on Qwen2-VL reaches 63.4% on LIBERO-90, while mixed training without separation falls to 50.5%, so naive mixing hurts even the embodied task.
Limitations
- Embodied scenario coverage is limited compared with GUI scenarios; only LIBERO is used.
- Inference assumes the task type (GUI or embodied) is known and routes to the corresponding branch.
Experimental setup
Datasets: OS-Atlas UGround Aguvis Aria-UI LIBERO AndroidControl GUI-Odyssey
Benchmarks: LIBERO-90 AndroidControl-Low AndroidControl-High GUI Odyssey
Metrics: success rate
Baselines: Claude 3.5 GPT-4o Qwen2-VL-7B UI-TARS-7B SeeClick Aria-UI OS-Atlas-7B Aguvis-7B ScaleTrack-7B MUTEX Distill-D ACT
Builds on: Qwen2-VL Qwen2.5-VL Magma GEA Aguvis
Closest papers by meaning
- MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World (CVPR 2024 · similarity 0.88)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.87)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.87)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.87)
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.87)
- ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.