Research map / Video / Action recognition
Understanding Video Transformers via Universal Concept Discovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VTCD, an unsupervised concept discovery method for video transformers that clusters feature-space tubelets, ranks concepts with CRIS, and mines Rosetta concepts shared across models.
Problem
Concept-based interpretability has only been studied for image models; explaining video transformers requires discovering high-level spatiotemporal concepts and ranking their importance, which existing image methods do not scale to or handle for transformer heads.
Approach
Generate tubelet proposals by SLIC clustering in the model's feature space, cluster them with convex NMF into concepts, rank concepts with CRIS (randomized parallel concept masking across all layers and heads), and identify universal Rosetta concepts shared across models via mIoU of concept supports.
Key results
- VTCD tubelets reduce positive-perturbation mIoU for TCOW from 0.174 (random crop + occlusion) to 0.102, and for VideoMAE from 0.240 to 0.094.
- Pruning 33% of VideoMAE heads (guided by CRIS) raises accuracy on six spill classes from 37.1 to 41.4 while cutting GFLOPs from 180.5 to 121.5.
- Concept-based VOS on DAVIS 2016 reaches 68.1 mIoU with VTCD + SAM on VideoMAE-SSL features, versus 45.0 without SAM refinement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune 33% of the least-important heads using VTCD | VideoMAE · Something-Something-v2, six spill classes | accuracy: 37.1 -> 41.4 | Helped |
| Prune 33% of the least-important heads using VTCD | VideoMAE · Something-Something-v2, six spill classes | GFLOPs: 180.5 to 121.5 | Helped |
| Prune 50% of the least-important heads using VTCD | VideoMAE · Something-Something-v2, six spill classes | accuracy: 37.1 -> 37.8 | Helped |
| Prune 50% of the least-important heads using VTCD | VideoMAE · Something-Something-v2, six spill classes | GFLOPs: 180.5 to 91.1 | Helped |
| Add VTCD object-tracking concepts and refine masks with SAM | VideoMAE-SSL features · DAVIS’16 validation | mIoU: 45.0 -> 68.1 | Helped |
| Add VTCD object-tracking concepts and refine masks with SAM | VideoMAE features · DAVIS’16 validation | mIoU: 43.1 -> 66.6 | Helped |
| Add VTCD object-tracking concepts and refine masks with SAM | InternVideo features · DAVIS’16 validation | mIoU: 45.8 -> 68.0 | Helped |
| Generate tubelet proposals with feature-space SLIC rather than random crops (ablation) | VTCD · TCOW Kubric | Positive perturbation: 0.174 -> 0.102 | Helped |
| Generate tubelet proposals with feature-space SLIC rather than random crops (ablation) | VTCD · TCOW Kubric | Negative perturbation: 0.274 -> 0.288 | Helped |
| Generate tubelet proposals with feature-space SLIC rather than random crops (ablation) | VTCD · VideoMAE on Something-Something-v2 | Positive perturbation: 0.240 -> 0.094 | Helped |
| Generate tubelet proposals with feature-space SLIC rather than random crops (ablation) | VTCD · VideoMAE on Something-Something-v2 | Negative perturbation: 0.300 -> 0.625 | Helped |
| Use CRIS instead of single-concept occlusion for concept importance (ablation) | VTCD · TCOW Kubric | Positive perturbation: 0.174 -> 0.166 | Helped |
Surprising findings
- Removing up to 70% of the least important concepts increases VideoMAE accuracy on SSv2, suggesting irrelevant concepts hurt predictions.
- Object-centric and hand-tracking concepts emerge in self-supervised VideoMAE despite never being trained for object-container reasoning.
Limitations
- Single concept masking has little effect on transformers, so the method requires many sampled concept sets (thousands) to rank importance.
- Segmentation accuracy from concept masks is limited by low mask resolution, needing SAM refinement.
- TCOW requires a manually labeled initial mask when applied to SSv2.
Experimental setup
Datasets: TCOW Kubric Something-Something-v2 DAVIS 2016
Benchmarks: Something-Something-v2 DAVIS 2016
Metrics: attribution curve mIoU classification accuracy GFLOPs mIoU R-score
Baselines: Baseline + Occ (random crop tubelets with occlusion importance) Integrated gradients Random concept ordering CRIS-based random crop baseline
Builds on: ACE CRAFT Lens (concept occlusion) Rosetta neurons
Closest papers by meaning
- Object-Region Video Transformers (CVPR 2022 · similarity 0.88)
- Long-Short Temporal Contrastive Learning of Video Transformers (CVPR 2022 · similarity 0.87)
- Self-Supervised Video Transformer (CVPR 2022 · similarity 0.87)
- Adapting Short-Term Transformers for Action Detection in Untrimmed Videos (CVPR 2024 · similarity 0.86)
- Compositional Video Understanding with Spatiotemporal Structure-based Transformers (CVPR 2024 · similarity 0.86)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.