Research map / Vision-language and multimodal / Cross-modal learning
MODUS: Decoder-only Any-to-Any Modeling of Diverse Modalities
Mingqiao Ye, Zhaochong An, Zhitong Gao, Xian Liu, Oğuzhan Fatih Kar, Jesse Allardice, Roman Bachmann, David Mizrahi, François Fleuret, Chuan Li, Amir Zadeh, Serge Belongie and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VAE features to ViT-only conditioning (ablation) | MODUS · NYUv2 surface normal | angular error: 35.30 -> 19.92 | Improved |
| add ViT features to VAE-only conditioning (ablation) | MODUS · NYUv2 depth | depth error: 6.9 -> 6.5 | Improved |
| use uniform instead of logit-normal timestep sampling (ablation) | MODUS 2D experts · NYUv2 surface normal | angular error: 50.54 -> 21.10 | Improved |
| use uniform instead of logit-normal timestep sampling (ablation) | MODUS 2D experts · NYUv2 depth | depth error: 21.9 -> 8.4 | Improved |
| use uniform instead of mode timestep sampling (ablation) | MODUS 2D experts · GenEval | GenEval score: 0.80 -> 0.81 | Improved |
| route surface normal estimation through canny edges (ablation) | MODUS · NYUv2 surface normal | angular error: 20.02 -> 19.87 | Improved |
| route surface normal estimation through DINOv2 features (ablation) | MODUS · NYUv2 surface normal | angular error: 20.02 -> 20.71 | Hurt |
| route surface normal estimation through depth (ablation) | MODUS · NYUv2 surface normal | angular error: 20.02 -> 20.06 | No clear effect |
| add VQA and grounding verifiers for self-verification (ablation) | MODUS text-to-image · GenEval | GenEval score: 0.81 -> 0.84 | Improved |
| use unified sequence instead of decoupled per-modality heads (ablation) | MODUS · NYUv2 surface normal | angular error: 50.75 -> 19.92 | Improved |
| use unified sequence instead of decoupled per-modality heads (ablation) | MODUS · DIODE depth | depth error: 0.322 -> 0.285 | Improved |
| use unified sequence instead of decoupled per-modality heads (ablation) | MODUS · MMMU | accuracy: 50.4 -> 51.1 | No clear effect |
Closest papers by meaning
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.89)
- 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities (NeurIPS 2024 · similarity 0.87)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.87)
- Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge (NeurIPS 2025 · similarity 0.87)
- ViT-Lens: Towards Omni-modal Representations (CVPR 2024 · similarity 0.85)
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
