Research map / Vision-language and multimodal / Human and embodied modeling
4M: Massively Multimodal Masked Modeling
David Mizrahi, Roman Bachmann, Oguzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, Amir Zamir
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train with all target modalities instead of RGB only (RGB inputs) (ablation) | 4M · average transfer loss | avg loss: 5.49 -> 5.29 | Improved |
| pre-train with all inputs and all targets instead of RGB inputs (avg loss) (ablation) | 4M · average transfer loss | avg loss: 5.29 -> 5.36 | Mixed |
| use multimodal inputs (All) vs RGB-only inputs on X->Y transfers (ablation) | 4M · X->Y transfer loss | cross-entropy: 6.91 -> 6.75 | Improved |
| use multimodal inputs vs RGB-only on RGB->X transfers (ablation) | 4M · RGB->X transfer loss | cross-entropy: 4.74 -> 5.07 | Hurt |
| add captions as the only target (ablation) | 4M · average transfer loss | avg loss: 5.49 -> 6.03 | Hurt |
| multi-target CLIP pre-training (4M-B) vs RGB -> CLIP only (ablation) | 4M-B · COCO | AP box: 46.6 -> 49.7 | Improved |
| multi-modal multitask pre-training (4M-B) vs RGB -> RGB only (ablation) | 4M-B · NYU depth | delta1 accuracy: 80.4 -> 92.0 | Improved |
| 4M-B vs MAE B | 4M-B · COCO | AP box: 48.3 -> 49.7 | Improved |
| 4M-B vs DeiT III B | 4M-B · ImageNet-1K | top-1 accuracy: 85.4 -> 84.5 | Hurt |
| 4M-L vs MAE L | 4M-L · NYU depth | delta1 accuracy: 93.6 -> 94.4 | Improved |
| increase Dirichlet alpha (uniform simplex sampling) (ablation) | 4M · avg transfer loss | avg loss | Improved |
| decode only a small random subset of targets (target masking) (ablation) | 4M · avg transfer loss | avg loss | Improved |
Closest papers by meaning
- 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities (NeurIPS 2024 · similarity 0.92)
- Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.89)
- Instruction-Guided Visual Masking (NeurIPS 2024 · similarity 0.88)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.87)
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
