Research map / Video / Video segmentation and tracking
VideoMAE V2: Scaling Video Masked Autoencoders With Dual Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add decoder masking (running cell, 50%, loss on encoder-invisible tokens) on top of encoder tube masking (ablation) | VideoMAE ViT-B pre-training · Something-Something V2 | Top-1 accuracy: 70.28 -> 70.15 | No clear effect |
| Replace encoder-only masking with dual masking (ViT-B) (ablation) | VideoMAE ViT-B · Something-Something V2 pre-training cost | FLOPs, memory, time: 35.48G->25.87G FLOPs; 631M->328M mem; 1.79x speedup | Helped |
| Replace encoder-only masking with dual masking (ViT-g) (ablation) | VideoMAE ViT-g on UnlabeledHybrid · UnlabeledHybrid pre-training cost | Running time and memory: 356h->241h (1.48x); 1753M->1050M | Helped |
| Use frame-level decoder masking (reconstruct half frames) instead of no decoder masking (ablation) | VideoMAE ViT-B decoder · Something-Something V2 | Top-1 accuracy: 70.28 -> 69.76 | Hurt |
| Use random cube decoder masking (50%) instead of no decoder masking (ablation) | VideoMAE ViT-B decoder · Something-Something V2 | Top-1 accuracy: 70.28 -> 64.87 | Hurt |
| Compute reconstruction loss over all decoder output tokens (running cell 50%) rather than only encoder-invisible tokens (ablation) | dual masking decoder · Something-Something V2 | Top-1 accuracy: 70.15 -> 66.74 | Hurt |
| Pre-train on UnlabeledHybrid (1.35M clips) instead of small single dataset with VideoMAE | ViT-B/L/H/g VideoMAE V2 · Kinetics-400 and Something-Something V2 | Top-1 accuracy: ViT-H SSv2 +2.0 over VideoMAE V1; ViT-g K400 +0.3 at ViT-H level | Helped |
| Add supervised K710 post-pre-training stage before fine-tuning (ViT-H) (ablation) | VideoMAE V2 ViT-H · Kinetics-400 | Top-1 accuracy: 86.9 -> 88.6 | Helped |
| Add K710 post-pre-training to VideoMAE V1 ViT-H | VideoMAE V1 ViT-H · Kinetics-400 | Top-1 accuracy: 86.6 -> 88.1 | Helped |
| Add K710 post-pre-training stage on ViT-g (ablation) | VideoMAE V2 ViT-g · Kinetics-400 | Top-1 accuracy: 87.2 -> 88.5 | Helped |
| Post-pre-training on SSv2 (labeled hybrid) instead of K710 | VideoMAE V2 pre-trained models · Something-Something V2 | Top-1 accuracy | Hurt |
| Scale model from ViT-H to ViT-g on UnlabeledHybrid | VideoMAE V2 · Kinetics-400 and Something-Something V2 | Top-1 accuracy: K400 86.9 -> 87.2; SSv2 76.8 -> 77.0 (smaller gain than ViT-L to ViT-H) | Helped |
Closest papers by meaning
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.92)
- EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens (ICML 2024 · similarity 0.91)
- VideoMAC: Video Masked Autoencoders Meet ConvNets (CVPR 2024 · similarity 0.90)
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.90)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.89)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.