Research map / Vision-language and multimodal / Robot action and navigation
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unified-IO 2 trains a 1.1B-6.8B encoder-decoder transformer from scratch on multimodal data (text, image, audio, video, action) with a multimodal mixture-of-denoisers objective, then instruction-tunes on 120+ datasets to handle understanding and generation across modalities.
Problem
Building a single autoregressive model that can both understand and generate many modalities (image, text, audio, video, action) is hard to train stably and to instruct-tune across diverse tasks.
Approach
Tokenize all inputs and outputs (images, text, audio, boxes, keypoints, robot actions) into a shared token space processed by one encoder-decoder transformer, with architectural stabilizers (2D RoPE, QK normalization, scaled cosine attention in the perceiver), a multimodal mixture-of-denoisers pretraining objective, and dynamic packing for throughput.
Key results
- On the GRIT benchmark, UNIFIED-IO 2 XXL (6.8B) reaches 66.9 overall on the ablation set and 67.0 on test, surpassing UNIFIED-IO by 2.7 points.
- On image generation, UNIFIED-IO 2 XXL scores 81.3 on TIFA, about 10 points ahead of CoDi and Emu.
- On VIMA-Bench robot manipulation, UNIFIED-IO 2 XXL achieves a 56.3% average success rate versus 72.6% for the original VIMA policy and 42.5% for VIMA-IMG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase model size from 1.1B to 6.8B parameters (ablation) | UNIFIED-IO 2 · GRIT ablation set | All: 60.8 -> 66.9 | Helped |
| increase model size from 3.2B to 6.8B parameters (ablation) | UNIFIED-IO 2 · GRIT ablation set | All: 65.2 -> 66.9 | Helped |
| use a unified model instead of UIOXL | GRIT test · GRIT test | All: 64.3 -> 67.0 | Helped |
| use UNIFIED-IO 2 instead of other generalist image-generation models | text-to-image generation · TIFA | score: 71.6 -> 81.3 | Helped |
| use UNIFIED-IO 2 instead of compared image-generation models | text-to-image generation · MS COCO | FID: 13.39 | Hurt |
| use UNIFIED-IO 2 instead of AudioLDM-L | text-to-audio generation · AudioCaps | FAD: 1.96 -> 2.64 | Hurt |
| use UNIFIED-IO 2 instead of VIMA | robot manipulation · VIMA-Bench | success rate: 72.6 -> 56.3 | Hurt |
| use UNIFIED-IO 2 instead of LLaVa-1.5 (13B) | vision-language models · VQAv2 | score: 80.0 -> 79.4 | Hurt |
| use UNIFIED-IO 2 instead of MBT | audio classification · Kinetics-Sounds | score: 85.0 -> 89.3 | Helped |
| fine-tune specifically for depth estimation | UNIFIED-IO 2 · NYUv2 depth | RMSE: 0.623 -> 0.423 | Helped |
| use dynamic packing | UNIFIED-IO 2 training · multimodal training | training throughput: 4x | Helped |
Surprising findings
- Weak depth estimation: RMSE of 0.623 on NYUv2 zero-shot, improving to 0.423 only after task-specific fine-tuning.
- Language modeling lags dedicated LLMs on HellaSwag (54.3 for XXL vs. 76.1 for LLaMA-7B), attributed to far fewer training tokens (~250B).
Limitations
- Weaker OK-VQA performance due to limited text pretraining compared to dedicated language models.
- Weak depth estimation and object detection in crowded scenes.
- Audio generation limited to 4.08-second clips, requiring non-standard evaluation against AudioCaps.
Experimental setup
Datasets: LAION-400M CC3M CC12M RedCaps OBELICS YT-Temporal-1B ACAV100M AudioSet WebVid-10M HD-VILA-10M Ego4D Objaverse
Benchmarks: GRIT HellaSwag TIFA SEED-Bench AudioCaps MS COCO VIMA-Bench VQAv2 OK-VQA ScienceQA TallyQA RefCOCO
Metrics: average score accuracy FID TIFA FAD IS KL success rate CIDEr RMSE
Baselines: Unified-IO CoDi Emu Qwen-VL LLaVA-1.5 mPLUG-Owl2 InstructBLIP VIMA
Builds on: Unified-IO UL2 mixture of denoisers ViT-VQGAN Perceiver Resampler RoPE
Closest papers by meaning
- M*: A Modular, Extensible, Serving System for Multimodal Models (NeurIPS 2026 · similarity 0.87)
- Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.87)
- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.86)
- Show-o2: Improved Native Unified Multimodal Models (NeurIPS 2025 · similarity 0.86)
- Multimodal Autoregressive Pre-training of Large Vision Encoders (CVPR 2025 · similarity 0.86)
- CoDi-2: In-Context Interleaved and Interactive Any-to-Any Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.