Research map / Video / Video segmentation and tracking
SIGMA: Sinkhorn-Guided Masked Video Modeling
Mohammadreza Salehi, Michael Dorkenwald, Fida Mohammad Thoker, Efstratios Gavves, Cees Snoek, Yuki M Asano
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace pixel reconstruction targets with Sinkhorn-clustered learned features (MLP projection) | VideoMAE · UCF-101 linear probe | top-1 accuracy (%): 58.6 -> 73.8 | Improved |
| use frozen DINO as projection network (ablation) | SIGMA · K400 linear probe | top-1 accuracy (%): 30.7 -> 47.5 | Improved |
| use L2 loss on learned deep targets instead of Sinkhorn cluster prediction (ablation) | SIGMA-MLP · mini SSv2 (ViT-S) | top-1 accuracy (%): 60.3 -> 36.4 | Hurt |
| increase number of prototypes from 1000 to 4000 (ablation) | SIGMA-MLP · mini SSv2 (ViT-S) | top-1 accuracy (%): 59.7 -> 60.3 | Improved |
| increase prototypes from 4000 to 6000 (ablation) | SIGMA-MLP · mini SSv2 (ViT-S) | top-1 accuracy (%): 60.3 -> 60.1 | No clear effect |
| change projection MLP depth or width (ablation) | SIGMA-MLP · mini SSv2 (ViT-S) | top-1 accuracy (%): 59.7 -> 59.2 (shallower), 59.6 (wider) | No clear effect |
| replace VideoMAE pixel targets with SIGMA-MLP in full finetuning on K400 | VideoMAE ViT-B · K400 full finetuning | top-1 accuracy (%): 80.0 -> 80.2 | Improved |
| use SIGMA-DINO versus SVT on multi-label Charades | SIGMA · SEVERE Charades | score: 35.5 (SVT) vs 23.3 | Hurt |
| use SIGMA features for unsupervised video object segmentation | VideoMAE · YTVOS overclustering ViT-B | mIoU: 49.7 -> 60.4 (SIGMA-DINO) | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.
- VideoMAE — VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (NeurIPS 2022) · better on 7 of 7 shared results
- MGMAE — MGMAE: Motion Guided Masking for Video Masked Autoencoding (ICCV 2023) · better on 5 of 5 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.88)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.88)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.87)
- VideoMAC: Video Masked Autoencoders Meet ConvNets (CVPR 2024 · similarity 0.87)
- EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens (ICML 2024 · similarity 0.87)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
