AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

MultiMAE: Multi-modal Multi-task Masked Autoencoders

Roman Bachmann, David Mizrahi, Andrei Atanov, Amir Zamir

ECCV 2022 · Vision-language and multimodal › Cross-modal learning · approach: Multi-modality image fusion · 334 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add ground-truth depth as extra input (RGB-D) at transferMultiMAE · Hypersim semantic segmentationmIoU: 37.0 -> 47.6Improved
add ground-truth depth as extra input (RGB-D) at transferMAE · Hypersim semantic segmentationmIoU: 36.5 -> 36.9No clear effect
add ground-truth depth as extra input (RGB-D) at transferMultiMAE · NYUv2 semantic segmentationmIoU: 52.0 -> 56.0Improved
add ground-truth depth as extra input (RGB-D) at transferMAE · NYUv2 semantic segmentationmIoU: 50.8 -> 49.3Hurt
add pseudo-labeled depth and semantic segmentation inputsMultiMAE · NYUv2 semantic segmentationmIoU: 52.0 -> 54.0Improved
add pseudo-labeled depth and semantic segmentation inputsMultiMAE · Hypersim semantic segmentationmIoU: 37.0 -> 40.1Improved
add pseudo-labeled depth and semantic segmentation inputsMAE · NYUv2 semantic segmentationmIoU: 50.1 -> 49.3Hurt
add pseudo-labeled depth and semantic segmentation inputsMultiMAE · ADE20K semantic segmentationmIoU: 46.2 -> 47.1Improved
add semantic segmentation as pre-training task alongside RGB-depth (ablation)MultiMAE (400 epochs) · NYUv2 semantic segmentationmIoU: 45.8 -> 50.6Improved
add depth as pre-training task alongside RGB-semseg (ablation)MultiMAE (400 epochs) · NYUv2 semantic segmentationmIoU: 51.6 -> 50.6Hurt
add masking to multi-task RGB-depth-semseg pre-training (ablation)MultiMAE (400 epochs) · NYUv2 semantic segmentationmIoU: 48.6 -> 50.6Improved
use deeper decoder (depth 8 instead of depth 2) in MAE (ablation)MAE · ADE20KmIoU: 43.3 -> 46.2Improved

Later papers that compared with this method

1 later accepted paper reports MultiMAE as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy