Research map / Video / Video segmentation and tracking
SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMILE, a masked video modeling method that reconstructs CLIP features instead of pixels and overlays synthetic object motion trajectories on videos (with trajectory-based masking) to learn video representations, including without natural videos.
Problem
Masked video modeling relies on pixel reconstruction, which overfits low-level appearance and is weak at capturing high-level semantics and motion dynamics due to temporal redundancy in natural videos.
Approach
Replace pixel targets with frozen CLIP feature targets for masked tokens, and inject synthetic moving, rotated, scaled segmented objects along random smooth trajectories into videos, masking the object trajectories to force motion encoding.
Key results
- Linear probing on K400 with ViT-B reaches 56.2% top-1 (SMILE) vs 47.5% for SIGMA and 20.7% for VideoMAE.
- Full finetuning on Something-Something V2 reaches 72.5% top-1 with SSv2 pretraining, and 83.1% on K400 with K400 pretraining.
- Pretraining with only synthetic object motion on black or noise backgrounds (no natural videos) improves K400m accuracy from 39.7% (no pretraining) to 59.2-61.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add synthetic motion to pixel-target training (ablation) | SMILE · K400m | Top-1 accuracy (%): 66.0 -> 68.1 | Helped |
| add synthetic motion to feature-target training (ablation) | SMILE · SSv2m | Top-1 accuracy (%): 60.2 -> 61.6 | Helped |
| use CLIP features instead of MAE features as reconstruction targets (ablation) | SMILE · K400m | Top-1 accuracy (%): 77.6 -> 78.1 | Helped |
| add both rotation and scaling transformations to synthetic objects (ablation) | SMILE · K400m | Top-1 accuracy (%): 77.0 -> 78.9 | Helped |
| use progressive training instead of single-stage augmented-video training (ablation) | SMILE · K400m | Top-1 accuracy (%): 78.9 -> 79.8 | Helped |
| reduce masking ratio from 95% to 80% (ablation) | SMILE · K400m | Top-1 accuracy (%): 77.7 -> 80.3 | Helped |
| use two synthetic objects instead of one (ablation) | SMILE · K400m | Top-1 accuracy (%): 78.6 -> 78.9 | Helped |
| increase synthetic objects from two to three (ablation) | SMILE · K400m | Top-1 accuracy (%): 78.9 -> 78.6 | Hurt |
| add trajectory-based masking for synthetic object motion (ablation) | SMILE · K400m | Top-1 accuracy (%): 77.8 -> 78.9 | Helped |
| replace CLIP-feature reconstruction with pixel reconstruction (ablation) | SMILE · K400m | Top-1 accuracy (%): 78.1 -> 66.0 | Hurt |
| use CLIP-feature reconstruction instead of pixel reconstruction (ablation) | SMILE · SSv2m | Top-1 accuracy (%): 55.1 -> 60.2 | Helped |
| use SMILE rather than SIGMA for full finetuning | ViT-B pretrained on K400 · Kinetics-400 | Top-1 accuracy (%): 81.5 -> 83.1 | Helped |
Surprising findings
- Synthetic object motion alone on black or noise backgrounds, with no natural videos, yields strong video representations.
Limitations
- Synthetic motions may not appear realistic, as acknowledged by the authors.
- Domain shift from synthetic motion augmentation requires progressive or mixed training to mitigate.
Experimental setup
Datasets: Kinetics-400 SomethingSomething V2 Places UCF-101 HMDB-51 FineGYM EPIC-Kitchens-100 Charades SEVERE benchmark
Benchmarks: Kinetics-400 SomethingSomething V2 UCF-101 HMDB-51 FineGYM EPIC-Kitchens-100 Charades SEVERE
Metrics: Top-1 accuracy
Baselines: VideoMAE MVD MME MGMAE EVEREST MGM SIGMA OmniMAE ViCLIP UMT MotionMAE SVT
Builds on: VideoMAE MGMAE MGM PIN CLIP
Closest papers by meaning
- Masked Motion Encoding for Self-Supervised Video Representation Learning (CVPR 2023 · similarity 0.88)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.86)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.85)
- Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation (CVPR 2025 · similarity 0.85)
- HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding (NeurIPS 2026 · similarity 0.85)
- No More Shortcuts: Realizing the Potential of Temporal Self-Supervision (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.