Research map / Vision-language and multimodal / Robot action and navigation
Magma: A Foundation Model for Multimodal AI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Magma pretrains a multimodal foundation model on images, videos, UI and robotics data with Set-of-Mark and Trace-of-Mark surrogate tasks to enable action grounding and planning across digital and physical environments.
Problem
Existing vision-language-action models specialize in actions at the cost of general multimodal understanding, and simply mixing verbal-heavy understanding data with spatial action data does not help; a unified agent foundation model is needed.
Approach
Convert actions into visual-spatial surrogate targets: Set-of-Mark labels actionable regions in images for action grounding, and Trace-of-Mark predicts object motion traces from videos (with homography compensation for camera motion) for action planning, letting heterogeneous unlabeled and labeled data share one interface.
Key results
- Zero-shot on SimplerEnv, Magma surpasses the second-place OpenVLA by 20.8% and nearly doubles average success rate.
- On Mind2Web after finetuning, Magma-8B reaches 45.4% cross-website step success rate versus 31.0% for Qwen2-UIX and 30.1% for GPT-4-MindAct.
- Ablation: adding SoM+ToM raises VisualWebBench element grounding from 90.1 to 96.3 and SimplerEnv Google Robot from 37.5 to 52.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SoM and ToM to full multimodal pretraining (ablation) | Magma-8B · ScreenSpot Overall | score: 57.4 -> 61.4 | Helped |
| add SoM and ToM to full multimodal pretraining (ablation) | Magma-8B · VisualWebBench | element accuracy: 90.1 -> 96.3 | Helped |
| add SoM and ToM to full multimodal pretraining (ablation) | Magma-8B · VisualWebBench | action grounding: 25.2 -> 71.8 | Helped |
| add SoM and ToM to full multimodal pretraining (ablation) | Magma-8B · SimplerEnv Bridge | success rate: 17.7 -> 35.4 | Helped |
| add SoM and ToM to full multimodal pretraining (ablation) | Magma-8B · SimplerEnv Google Robot | success rate: 37.5 -> 52.3 | Helped |
| combine UI and robotics data without SoM and ToM instead of UI-only training (ablation) | Magma-8B · ScreenSpot Overall | score: 57.7 -> 56.2 | Hurt |
| add video data to pretraining (ablation) | Magma-8B · BLINK | accuracy: ∼8% performance drop | Helped |
| add SoM and ToM pretraining (ablation) | Magma-8B · VSR | accuracy: 58.1 -> 65.1 | Helped |
| add SoM and ToM pretraining (ablation) | Magma-8B · BLINK-val | accuracy: 38.3 -> 41.0 | Helped |
Surprising findings
- Simply combining UI and robotics data hurts performance on both tasks, and adding video narrations only slightly helps without SoM/ToM.
- Magma's pretraining lacks a post-training stage, so it follows instructions poorly and benefits from standard option matching in spatial benchmarks.
Limitations
- No post-training/instruction tuning stage, so instruction following is weak.
- Robot evaluation on real hardware only after finetuning on about 50 trajectories per task with few trials.
Experimental setup
Datasets: ShareGPT4V LLaVA-1.5 instruction data OCR datasets Open X-Embodiment SeeClick Vision2UI Ego4D Epic-Kitchens Something-Something v2 Mind2Web AITW Instructional videos
Benchmarks: VisualWebBench ScreenSpot SimplerEnv VQAv2 TextVQA POPE Mind2Web AITW VSR BLINK SpatialEval IntentQA
Metrics: success rate element accuracy step success rate accuracy F1
Baselines: GPT-4V GPT-4V-OmniParser LLaVA-1.5 LLaVA-Next Qwen-VL SeeClick OpenVLA Octo RT-1-X CogAgent
Builds on: Set-of-Mark prompting OpenVLA SeeClick CoTracker LLaVA
Closest papers by meaning
- DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.89)
- Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (ICLR 2026 · similarity 0.88)
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.88)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.88)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.88)
- BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.