Research map / Vision-language and multimodal / Human and embodied modeling
4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a single any-to-any masked multimodal Transformer (4M-21, up to 3B params) on 21 modalities spanning RGB, geometric, semantic, edge, feature-map, metadata and text, via discrete tokenization and co-training on CC12M, COYO700M and C4.
Problem
Existing any-to-any multimodal/multitask models such as 4M and Unified-IO cover only a small number of modalities and tasks; it is unclear whether scaling to tens of diverse modalities degrades performance.
Approach
Extend the 4M multimodal masked modeling scheme to 21 modalities by adding pseudo-labeled SAM instances, 3D human poses, color palettes, metadata, DINOv2/ImageBind/CLIP features, and T5-XXL embeddings, all mapped to discrete tokens with modality-specific tokenizers, and co-train across multiple datasets including text-only C4.
Key results
- 4M-21 XL matches 4M-7 XL on out-of-the-box normals (20.8 vs 20.6), depth (0.68 vs 0.69), and segmentation (48.1 vs 48.1) while solving 3x more tasks.
- On ImageNet-1K transfer, 4M-21 XL reaches 87.1% top-1 and on ARKitScenes 3D detection reaches 48.4 AP3D, the best among compared models.
- Ensembling 10 sampled predictions improves DIODE normals and COCO semseg for 4M-21 XL (20.8 vs 21.3 without ensemble; 48.1 vs 46.5 mIoU).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train 4M-21 XL on 21 modalities rather than 4M-7 XL on 7 modalities | 4M · DIODE | mean angle error: 20.6 -> 20.8 | No clear effect |
| Train 4M-21 XL on 21 modalities rather than 4M-7 XL on 7 modalities | 4M · COCO | semantic segmentation mIoU: 48.1 -> 48.1 | No clear effect |
| Train on 21 modalities rather than 7 modalities | 4M XL · ARKitScenes | AP3D: 48.1 -> 48.4 | Helped |
| Use RGB-D inputs rather than RGB-only inputs | 4M-21 XL · Hypersim | mIoU: 48.6 -> 52.5 | Helped |
| Use RGB-D inputs rather than RGB-only inputs | 4M-21 XL · ARKitScenes | AP3D: 48.4 -> 51.3 | Helped |
| Add C4 co-training to CC12M (ablation) | 4M-21 B* · ImageNet-1K | Top-1 accuracy: 84.4 -> 84.6 | Helped |
| Add C4 co-training to CC12M (ablation) | 4M-21 B* · NYUv2 | delta_1 accuracy: 90.9 -> 90.4 | Hurt |
| Add COYO700M co-training to CC12M+C4 (ablation) | 4M-21 B* · ARKitScenes | AP3D: 41.2 -> 42.4 | Helped |
| Ensemble ten predictions rather than use a single prediction (ablation) | 4M-21 · DIODE | mean angle error: 22.3 -> 21.7; 21.7 -> 21.1; 21.3 -> 20.8 | Helped |
| Ensemble ten predictions rather than use a single prediction (ablation) | 4M-21 · COCO | mean IoU: 39.0 -> 42.5; 43.8 -> 46.4; 46.5 -> 48.1 | Helped |
Surprising findings
- Co-training with 21 modalities and C4 text did not degrade performance versus specialist 7-modality 4M models, contrary to typical multitask negative transfer expectations.
- The paper reports that transfer/emergent capabilities to novel tasks remain largely untapped despite out-of-the-box gains.
Limitations
- Transfer or emergent capabilities to novel tasks remain largely untapped.
- Tokenizer reconstruction error bounds performance; better tokenizers needed.
- Co-training uses only partially aligned datasets (C4 text with CC12M modalities).
Experimental setup
Datasets: CC12M COYO700M C4 DIODE COCO ImageNet-1K 3DPW ADE20K NYUv2 ARKitScenes Hypersim
Benchmarks: DIODE COCO ImageNet-1K 3DPW ADE20K NYUv2 ARKitScenes Hypersim
Metrics: mean angle error L1 depth error mIoU k-NN accuracy AP3D top-1 accuracy delta1 accuracy
Baselines: 4M-7 Unified-IO Unified-IO 2 DINOv2 MAE DeiT III MultiMAE Omnidata Mask2Former SAM
Builds on: 4M Unified-IO MultiMAE VQ-VAE DINOv2
Closest papers by meaning
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.88)
- Task Me Anything (NeurIPS 2024 · similarity 0.88)
- How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks (ICLR 2026 · similarity 0.88)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.87)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.