Research map / Video / Video segmentation and tracking
Space-Time Crop & Attend: Improving Cross-Modal Video Representation Learning
Mandela Patrick, Po-Yao Huang, Ishan Misra, Florian Metze, Andrea Vedaldi, Yuki M. Asano, João F. Henriques
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace two RGB crops with feature crops (ablation) | STiCA · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 58.6 -> 60.4 | Improved |
| add feature crops in space (ablation) | avg-pooled baseline · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 54.0 -> 59.9 | Improved |
| add more feature crops in space and time (ablation) | STiCA · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 59.9 -> 60.4 | Improved |
| add temporal feature crops (2x3 + 1x2 sizes only, fewer spatial) (ablation) | STiCA · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 59.9 -> 58.4 | Hurt |
| use transformer pooling in both pretraining and finetuning (ablation) | avg pooling · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 54.0 -> 60.3 | Improved |
| transformer pooling only at finetuning (ablation) | avg-pooled pretraining · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 54.0 -> 54.6 | No clear effect |
| transformer pooling only at pretraining (ablation) | avg-pooled finetuning · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 54.0 -> 52.1 | Hurt |
| increase transformer layers from 2 to 4 (ablation) | STiCA · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 60.3 -> 58.3 | Hurt |
| add 2-layer transformer versus parameter-matched model (ablation) | R(2+1)D-18 · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 57.3 -> 60.3 | Improved |
| add transformer pooling to space+time feature crops (ablation) | STiCA · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 60.4 -> 62.0 | Improved |
| add cross-modal loss to within-modal baseline (ablation) | SimCLR-style video · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 43.3 -> 54.0 | Improved |
| add within-modal loss to cross-modal loss (ablation) | STiCA · HMDB-51 fold 1 (ablation, 100 epochs, 30 frames) | top-1 accuracy (%): 54.0 -> 58.6 | Improved |
Closest papers by meaning
- Spatio-Temporal Crop Aggregation for Video Representation Learning (ICCV 2023 · similarity 0.90)
- Spatiotemporal Contrastive Video Representation Learning (CVPR 2021 · similarity 0.89)
- A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (CVPR 2021 · similarity 0.88)
- Mosaic Representation Learning for Self-supervised Visual Pre-training (ICLR 2023 · similarity 0.87)
- Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics (CVPR 2019 · similarity 0.87)
- Contextualized Spatio-Temporal Contrastive Learning With Self-Supervision (CVPR 2022 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
