Research map / Video / Video segmentation and tracking
Structured-Noise Masked Modeling for Video, Audio and Beyond
Aritra Bhowmik, Carlos Hinojosa, Fida Mohammad Thoker, Bernard Ghanem, Cees Snoek
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace random tube masking with Green3D noise masking | VideoMAE · SSv2 (SSv2 pretraining) | top-1 accuracy: 69.6 -> 70.8 | Improved |
| add Green3D masking | SIGMA · SSv2 (SSv2 pretraining) | top-1 accuracy: 71.2 -> 72.0 | Improved |
| add Green3D masking | VideoMAE · DAVIS clustering | mIoU: 29.5 -> 38.2 | Improved |
| add Green3D masking | SIGMA · SEVERE benchmark (mean) | mean score: 64.2 -> 67.3 | Improved |
| replace random masking with R-BN masking | AudioMAE · AudioSet-20K | mAP: 36.1 -> 36.8 | Improved |
| replace random masking with R-BN masking | MaskSpec · AudioSet-20K | mAP: 32.3 -> 33.4 | Improved |
| use Green3D and R-BN masking in audio-visual pretraining | CAV-MAE · VGGSound | accuracy: audio-only (+0.6%), video-only (+0.8%), and audio-visual (+0.6%) | Improved |
| use Green noise rather than Blue or Red noise for video masks (ablation) | VideoMAE · mini-SSv2 | top-1 accuracy: 52.1 -> 54.5 | Improved |
| use Blue noise rather than Green for audio masks (ablation) | AudioMAE · AS-20k | mAP: 36.4 -> 36.5 | Improved |
| add regularization to blue noise (R-BN vs Blue) (ablation) | AudioMAE · AS-20k | mAP: 36.5 -> 36.8 | Improved |
| use 3D Green noise instead of per-frame Green2D masks (ablation) | VideoMAE · mini-SSv2 | top-1 accuracy: 52.9 -> 54.5 | Improved |
| use Green3D instead of grid or block masking (ablation) | VideoMAE · mini-Kinetics | top-1 accuracy: 51.0 -> 52.7 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- VideoMAE — VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (NeurIPS 2022) · better on 4 of 4 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation (ICLR 2025 · similarity 0.85)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.85)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.85)
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.85)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.85)
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
