Research map / Video / Video-language understanding
Extending Video Masked Autoencoders to 128 frames
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LVMAE, a video masked autoencoder that uses a jointly learned adaptive FSQ-MAGVIT tokenizer and token-importance scorer to choose which tokens to decode, enabling 128-frame MAE pre-training with reconstruction targets as quantized tokens.
Problem
Video MAE pre-training is typically limited to 16-32 frames because decoder self-attention over all masked tokens is memory-intensive and scales poorly with video length.
Approach
A content-dependent adaptive decoder mask selects top-k salient tokens (salience from a jointly trained FSQ-MAGVIT tokenizer with a CNN token scorer and differentiable top-k), plus a small random set; reconstruction targets are the quantized tokenizer latents rather than RGB. Decoding only ~15% of tokens makes 128-frame pre-training feasible.
Key results
- LVMAE (ViT-B, 128 frames) reaches 47.3 on EK100 verbs-noun-verb action accuracy with adaptive decoder masking, versus 44.5 for no decoder masking at 32 frames.
- On Diving48, LVMAE (ViT-L, unlabeled K710 pre-training) reaches 94.9 top-1, surpassing the prior SOTA MC-ViT-L at 91.0 by 3.9 points.
- On EK100 verbs, LVMAE (ViT-L, K710) reaches 75.0 vs AVION (ViT-B) baseline; the paper reports a +2.5 point improvement over SOTA on verb classification.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 128-frame rather than 32-frame pre-training | LVMAE · EPIC-Kitchens-100, 128-frame fine-tuning and evaluation | Top-1 action classification: 45.0 -> 47.3 | Helped |
| Use 128-frame rather than 32-frame pre-training | LVMAE · Diving48, 128-frame fine-tuning and evaluation | Top-1: 83.2 -> 87.9 | Helped |
| Use Adaptive instead of Random, Uniform, Flow, or EVEREST decoder masking | MAE with FSQ-MagViT targets · EPIC-Kitchens-100, 32 frames | Top-1 action classification: 42.20 ± 0.07, 41.73 ± 0.05, 42.18 ± 0.16, 39.24 ± 0.08 -> 43.21 ± 0.09 | Helped |
| Use Adaptive instead of Random, Uniform, or Flow decoder masking (ablation) | LVMAE with Adaptive FSQ-MagViT targets · Diving48, 128 frames | Top-1: 86.3, 85.2, 86.3 -> 87.9 | Helped |
| Add random tokens to Adaptive-selected tokens (ablation) | MAE decoder masking · EPIC-Kitchens-100, 32 frames | Noun-Verb accuracy: 42.5 -> 43.2 | Helped |
| Replace RGB reconstruction targets with Standard MAGVIT targets (ablation) | MAE without decoder masking · EPIC-Kitchens-100, 32 frames | Noun-Verb accuracy: 39.9 -> 43.2 | Helped |
| Use Adaptive FSQ-MagViT targets instead of Standard MAGVIT targets without decoder masking (ablation) | MAE · EPIC-Kitchens-100, 32 frames | Noun-Verb accuracy: 43.2 -> 42.6 | Hurt |
| Add Adaptive decoder masking with Adaptive FSQ-MagViT targets (ablation) | MAE · EPIC-Kitchens-100, 32 frames | Noun-Verb accuracy: 42.6 -> 43.2 | Helped |
| Increase context from 16 to 128 frames (ablation) | LVMAE · EPIC-Kitchens-100 | Noun-Verb accuracy: 41.7 -> 47.3 | Helped |
| Increase decoder budget from 15% to 50% (ablation) | LVMAE with Adaptive masking · Diving48, 128 frames | Top-1: 89.7 -> 87.5 | Hurt |
| Use FSQ with an 18-bit codebook rather than LFQ with a 14-bit codebook (ablation) | MAGVIT tokenizer and MAE · EPIC-Kitchens-100 | Top-1: 46.3 -> 46.4 | No clear effect |
| Increase context from 64 to 96 frames (ablation) | LVMAE · Something-Something-V2 | Top-1 accuracy: 71.0 -> 70.6 | Hurt |
Surprising findings
- Decoding only 15% of tokens with adaptive masking outperforms decoding all tokens (no decoder mask) at 32 frames when using FSQ-MagViT targets (43.2 vs 42.6).
- Random-only decoder masking at 15% budget underperforms adaptive masking, and fine-tuning gains saturate beyond 64 frames on EK100 (47.1 at 64 vs 47.3 at 128).
Limitations
- Experiments restricted to relatively small datasets and model sizes; no large-scale pre-training explored.
- Noun classification lags behind methods using large labeled or language-supervised pre-training data.
- Longer context gains diminish beyond 64 frames on EK100 due to few long videos in the data.
Experimental setup
Datasets: Kinetics-600 Kinetics-710 EPIC-Kitchens-100 Diving48 Something-Something-V2 FineGym288
Benchmarks: EPIC-Kitchens-100 Diving48 Something-Something-V2 FineGym288
Metrics: top-1 accuracy
Baselines: VideoMAE (no decoder mask) EVEREST Optical flow masking AVION MTV-B TAdaFormer MC-ViT LaViLa MeMViT TCT
Builds on: VideoMAE VideoMAEv2 dual masking MAGVIT FSQ EVEREST
Closest papers by meaning
- EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens (ICML 2024 · similarity 0.91)
- VideoMAC: Video Masked Autoencoders Meet ConvNets (CVPR 2024 · similarity 0.90)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.90)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.89)
- Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · similarity 0.89)
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.