Research map / Video / Video segmentation and tracking
VideoMAC: Video Masked Autoencoders Meet ConvNets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoMAC, a masked video autoencoder built on pure ConvNets (ResNet, ConvNeXtV2) with sparse convolution, symmetric masking on frame pairs, an online/EMA target dual-encoder, and a reconstruction consistency loss, pre-trained on YouTube-VOS.
Problem
Masked video modeling has relied on resource-intensive isotropic ViT encoders, which lack hierarchical local feature modeling for dense tasks, and ConvNets remain understudied for MVM because dense convolutions leak information into masked regions.
Approach
Use sparse convolutional ConvNet encoders (via MinkowskiEngine) to avoid mask dissipation, symmetric masking of consecutive frame pairs, an online and EMA target encoder-decoder pair, and a reconstruction consistency loss that aligns the reconstructions of the two frames to capture temporal correlation.
Key results
- VideoMAC (CNXv2-S) reaches 68.4 J&Fm on DAVIS17, 6.4 points above SiamMAE (62.0), and RN50 reaches 67.2 (+5.2).
- On VIP body part propagation, VideoMAC RN50 gets 43.6 mIoU vs 37.3 for SiamMAE (+6.3); on JHMDB, CNXv2-S gets 58.1 PCK@0.1 (+10.2 over SiamMAE's 47.0 in the table).
- Fine-tuned on ImageNet-1K, VideoMAC-pretrained CNXv2-T reaches 82.9% top-1, +0.4 over training from scratch (82.5%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoMAC with CNXv2-S instead of SiamMAE | Video representation learning · DAVIS17 | J &Fm: 62.0 -> 68.4 | Helped |
| Use VideoMAC with RN50 instead of SiamMAE | Video representation learning · VIP | mIoU: 37.3 -> 43.6 | Helped |
| Use VideoMAC with CNXv2-S instead of SiamMAE | Video representation learning · JHMDB | PCK@0.1: 47.0 -> 58.1 | Helped |
| Replace hierarchical encoder with isotropic CNXv2-S (ablation) | VideoMAC on DAVIS17 · DAVIS17 | J &Fm: 68.4 -> 53.8 | Hurt |
| Use asymmetric instead of symmetric masking (ablation) | VideoMAC on DAVIS17 · DAVIS17 | J &Fm: 68.4 -> 66.5 | Hurt |
| Use 2, 4, or 8 decoder blocks instead of 1 (ablation) | VideoMAC on DAVIS17 · DAVIS17 | J &Fm: 68.4 -> 68.2; 68.3; 68.1 | No clear effect |
| Set decoder width to 128, 256, or 1024 instead of 512 (ablation) | VideoMAC on DAVIS17 · DAVIS17 | J &Fm: 68.4 -> 67.9; 68.2; 68.1 | No clear effect |
| Add target reconstruction loss and reconstruction consistency loss (ablation) | VideoMAC on DAVIS17 · DAVIS17 | J &Fm: 51.1 -> 68.4 | Helped |
| Add reconstruction consistency loss to target reconstruction loss (ablation) | VideoMAC on DAVIS17 · DAVIS17 | J &Fm: 60.6 -> 68.4 | Helped |
| Increase consistency-loss weight γ from 0.0 to 0.1 or 0.5 (ablation) | VideoMAC on DAVIS17 · DAVIS17 | J &Fm: 60.6 -> 64.8; 65.8 | Helped |
| Pre-train on K400 instead of YT18 (ablation) | VideoMAC on DAVIS17 · DAVIS17 | J &Fm: 68.4 -> 67.8 | No clear effect |
| Use color augmentation, alone or with spatial augmentation (ablation) | VideoMAC on DAVIS17 · DAVIS17 | J &Fm: 68.4 -> 64.9; 66.1 | Hurt |
Surprising findings
- Pre-training on a 5.58-hour YouTube-VOS subset with ConvNets outperforms ViT-based MVM pre-trained on 800 hours of Kinetics-400.
- Using dense convolution (non-sparse) in ConvNet MVM causes the mask structure to dissipate, and asymmetric masking degrades DAVIS17 performance from 68.4 to 66.5 J&F.
Limitations
- Fixed downsampling ratio in ConvNets gives a relatively fixed patch size and coarse feature resolution (7x7 at 224 input).
- Performance drops sharply at a 0.95 masking ratio due to overfitting to the pretext task.
- Image recognition gains are small (+0.2 to +0.4 points over scratch) and the video-to-image transfer is limited.
Experimental setup
Datasets: YouTube-VOS 2018 Kinetics-400 ImageNet-1K DAVIS17 VIP JHMDB
Benchmarks: DAVIS17 VIP JHMDB ImageNet-1K
Metrics: J&F mean Jm Fm mIoU PCK@0.1 PCK@0.2 top-1 accuracy
Baselines: MAE VideoMAE MAE-ST DropMAE SiamMAE SparK FCMAE DenseCL DINO Supervised ResNet-50
Builds on: MAE SparK FCMAE SiamMAE BYOL-style EMA target network
Closest papers by meaning
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.90)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.89)
- EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens (ICML 2024 · similarity 0.89)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.89)
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.87)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.