AI Paper Atlas

Research map / Label-efficient and robust learning / Self-supervised and contrastive

Audiovisual Masked Autoencoders

Mariana-Iuliana Georgescu, Eduardo Fonseca, Radu Tudor Ionescu, Mario Lucic, Cordelia Schmid, Anurag Arnab

ICCV 2023 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Masked autoencoder pretraining · 54 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
pretrain jointly on audio and video instead of audio-only MAE, for audiovisual finetuning (ablation)Audiovisual MAE · VGGSound audiovisual (ViT-Base)top-1 accuracy: 58.3 -> 63.5Improved
pretrain jointly instead of video-only MAE, for audiovisual finetuning (ablation)Audiovisual MAE · VGGSound audiovisual (ViT-Base)top-1 accuracy: 62.1 -> 63.5Improved
audiovisual pretraining for unimodal finetuning (ablation)Audiovisual MAE · VGGSound audio-only / video-onlytop-1 accuracy: 55.8 vs 55.7 audio; 48.5 vs 49.3 videoNo clear effect
use joint reconstruction objective instead of modality inpainting (ablation)Audiovisual MAE · VGGSound audiovisualtop-1 accuracy: 58.4 -> 62.2Improved
use mid-fusion or separate encoder instead of early fusion or shared (ablation)Audiovisual MAE · VGGSound audiovisualtop-1 accuracy: 62.2 -> 63.5Improved
use shared decoder instead of early or separate decoder (ablation)Audiovisual MAE · VGGSound audiovisualtop-1 accuracy: 63.3 -> 63.5Improved
scale backbone from ViT-Base to ViT-Large with self-supervised pretraining (ablation)Audiovisual MAE · VGGSound audiovisualtop-1 accuracy: 64.2 -> 65.0Improved
scale backbone to ViT-Large with supervised ImageNet-21K pretraining (ablation)MBT · VGGSound audiovisualtop-1 accuracy: 64.1 -> 61.4Hurt
pretrain longer on VGGSound (800 to 1200 epochs) (ablation)Audiovisual MAE · VGGSound audiovisualtop-1 accuracy: 65.0 -> 64.9No clear effect
pretrain longer (200 to 1200 epochs) (ablation)Audiovisual MAE · Epic Kitchens audiovisualaction accuracy: 41.8 -> 46.0Improved
pretrain on VGGSound instead of the larger AudioSet (ablation)Audiovisual MAE · Epic Kitchens audiovisualaction accuracy: 43.5 -> 45.5Improved
joint audiovisual pretraining instead of two separate unimodal MAEs initialising MBT (ablation)Audiovisual MAE · VGGSound audiovisualtop-1 accuracy: 63.3 -> 64.2Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy