Research map / Video / Video segmentation and tracking
EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EVEREST, a masked video autoencoder that selects informative spatiotemporal tokens via embedding distance between adjacent frames and samples informative frames online, cutting compute and memory for pre-training and fine-tuning.
Problem
Masked video autoencoders reconstruct randomly masked tokens and frames, wasting large amounts of computation and memory on uninformative or redundant regions, making video representation learning require large GPU clusters.
Approach
Select tokens whose embeddings differ most from the same-location token in the previous frame pair (redundancy-robust masking, no optical flow or HOG needed), process only those tokens, and sample frames with probability proportional to their count of selected tokens (information-intensive frame selection).
Key results
- On K400 with ViT-L, EVEREST reaches 82.1% top-1 vs 82.0% for VideoMAE while cutting pre-training GFLOPs by 26.8% and memory by 74.1% (164.1 GB vs 634.1 GB).
- On UCF101 and HMDB51 with ViT-B, EVEREST reports 93.4% and 68.1%, outperforming VideoMAE (91.3% and 62.6%).
- On Ego4D OSCC, EVEREST (visual only) reaches 76.2% accuracy, above Egocentric VLP (73.9%, visual+text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply redundancy-robust token selection | EVEREST with ViT-B · UCF101 | Top-1 accuracy: 91.3 -> 93.4 | Helped |
| Apply redundancy-robust token selection | EVEREST with ViT-B · HMDB51 | Top-1 accuracy: 62.6 -> 68.1 | Helped |
| Apply redundancy-robust token selection | EVEREST with ViT-B · SSv2 | Top-1 accuracy: 64.3 -> 64.6 | Helped |
| Use L2 rather than negative cosine or negative CKA distance to score tokens (ablation) | EVEREST pre-training · HMDB51 | accuracy(%): 33.53; 34.58; 42.22 -> 42.81 | Helped |
| Use pre-training masking ratio 0.3 instead of 0.5, 0.4, or 0.2 (ablation) | EVEREST · UCF101 | top-1: 90.91; 91.51; 91.56; 90.80 | Mixed |
| Use information-intensive frame selection with α=1.5 rather than no frame selection (ablation) | EVEREST pre-training on OSCC · Ego4D OSCC | Accuracy (%): 73.17 -> 73.85 | Helped |
| Vary frame selection ratio α among 1.3, 1.5, 1.8, and 2.0 (ablation) | EVEREST pre-training on OSCC · Ego4D OSCC | Accuracy (%): 73.54; 73.85; 73.79; 73.80 | Mixed |
| Stack 2 or 3 Conv3d embedding layers rather than use one (ablation) | EVEREST · HMDB51 | FT Acc. (%): 68.1 -> 66.7; 66.8 | Hurt |
| Increase input length from 16 to 24 frames (ablation) | EVEREST pre-training and fine-tuning · UCF101 | Acc. (%): 93.39 -> 94.42 | Helped |
| Select far-distance tokens in both pre-training and fine-tuning instead of near-distance tokens (ablation) | EVEREST · UCF101 | Fine-tuning accuracy (%): 60.93; 73.49; 75.60 -> 91.56 | Helped |
| Use redundancy-robust masking during fine-tuning instead of full-token fine-tuning (ablation) | VideoMAE-pretrained ViT-B · K400 | Acc. (%): 81.5 -> 81.6 | No clear effect |
| Fine-tune with EVEREST token selection instead of full tokens (ablation) | VideoMAE-pretrained ViT-L · K400 | Acc. (%): 85.2 -> 84.8 | Hurt |
Surprising findings
- Fine-tuning with only 60% of tokens (rho_pre=0.6) on UCF101 slightly outperforms using all tokens (91.56% vs 89.71% at rho=1.0), at about 55% of the GFLOPs.
- Selecting the farthest-distance (most changed) tokens works much better than selecting nearest-distance tokens; reversing the selection drops HMDB51 accuracy to 60.93%.
Limitations
- Pre-training uses 200 epochs for most scaled ViT experiments rather than the original long schedules, so comparisons are not at full VideoMAE budgets.
- Token selection uses an L2 distance on Conv3d embeddings; the paper reports that other distance functions (cosine, CKA) perform worse, and the choice of selection ratio rho_pre is dataset-dependent.
Experimental setup
Datasets: Kinetics-400 Something-Something V2 UCF101 HMDB51 Ego4D
Benchmarks: Kinetics-400 UCF101 HMDB51 Something-Something V2 Ego4D OSCC
Metrics: top-1 accuracy GFLOPs memory usage (GB) pre-training time
Baselines: VideoMAE ST-MAE MME MVD K-centered patch sampling VCOP CoCLR RSPNet Egocentric VLP VideoMAE-V2 running-cell masking
Builds on: VideoMAE ST-MAE masked image modeling
Closest papers by meaning
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.91)
- VideoMAE V2: Scaling Video Masked Autoencoders With Dual Masking (CVPR 2023 · similarity 0.91)
- VideoMAC: Video Masked Autoencoders Meet ConvNets (CVPR 2024 · similarity 0.89)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.86)
- CanvasMAR: Improving Masked Autoregressive Video Prediction With Canvas (NeurIPS 2026 · similarity 0.85)
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.