Research map / Video / Video segmentation and tracking
DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DejaVid, an encoder-agnostic method that converts videos into variable-length temporal sequences of embeddings from a frozen VideoMAE V2-g and classifies them with a learned, time-weighted DTW neural network with per-timestep, per-feature weights and centroids.
Problem
Large video transformer encoders average clip embeddings over time, discarding variable duration, chronological order, and temporal variation in feature importance; existing temporal methods require invasive architecture changes and retraining.
Approach
Represent a video as a temporal sequence of embeddings and classify it via a DTW-inspired neural network with learned positive per-timestep, per-feature weights and class centroids (DBA-initialized), reformulated as serial min-pooling and skip-connections with a custom CUDA kernel, dropping the diagonal transition for stability.
Key results
- Achieves 77.2% top-1 on Something-Something V2 (vs. 76.75% VideoMAE V2-g baseline) with +17.6M parameters.
- Achieves 89.1% top-1 on Kinetics-400 (vs. 88.4%) with +5.8M to +11.6M parameters.
- Achieves 88.6% top-1 on HMDB51 (vs. 88.1%) with +1.19M parameters; custom CUDA kernel gives 161x-750x per-epoch speedup over naive DTW.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DejaVid with learned centroids and randomly initialized weights | VideoMAE V2-g · Kinetics-400 | top-1 accuracy: 88.4 -> 88.9 | Helped |
| Add DejaVid with learned centroids and randomly initialized weights | VideoMAE V2-g · Something-Something V2 | top-1 accuracy: 76.75 -> 77.2 | Helped |
| Add DejaVid with learned centroids and randomly initialized weights | VideoMAE V2-g · HMDB51 | top-1 accuracy: 88.1 -> 88.6 | Helped |
| Learn centroids rather than freezing them (ablation) | DejaVid · Kinetics-400 | top-1 accuracy: 88.6 -> 89.1 | Helped |
| Learn centroids rather than freezing them (ablation) | DejaVid · Something-Something V2 | top-1 accuracy: 76.9 -> 77.2 | Helped |
| Learn centroids rather than freezing them (ablation) | DejaVid · HMDB51 | top-1 accuracy: 88.4 -> 88.6 | Helped |
| Learn temporal feature weights rather than freezing weights initialized to one (ablation) | DejaVid · Kinetics-400 | top-1 accuracy: 89.1 -> 88.9 | No clear effect |
| Learn temporal feature weights rather than freezing weights initialized to one (ablation) | DejaVid · Something-Something V2 | top-1 accuracy: 77.1 -> 77.2 | Helped |
| Learn temporal feature weights rather than freezing weights initialized to one (ablation) | DejaVid · HMDB51 | top-1 accuracy: 88.3 -> 88.6 | Helped |
| Enable diagonal transitions in DTW (ablation) | DejaVid · Kinetics-400 | top-1 accuracy: 89.1 -> 88.6 | Hurt |
| Enable diagonal transitions in DTW (ablation) | DejaVid · Something-Something V2 | top-1 accuracy: 77.2 -> 77.0 | Hurt |
| Disable fixed epoch centroids and weights for warping-path determination (ablation) | DejaVid · Kinetics-400 | top-1 accuracy: 89.1 -> 89.0 | Hurt |
Surprising findings
- Temporal supersampling of VideoMAE V2-g alone does not improve accuracy, and learned temporal weights give no extra benefit on Kinetics-400 beyond centroid finetuning.
- Disabling the DTW diagonal transition and freezing per-epoch centroids/weights for path computation improved stability and accuracy in ablations.
Limitations
- Gains are modest (0.5-0.7% top-1) over the VideoMAE V2-g baseline.
- Only evaluated with one SOTA encoder (VideoMAE V2-g); other encoders only in supplementary material.
- Learned temporal weights can overfit; no regularization explored for Kinetics-400.
Experimental setup
Datasets: Something-Something V2 Kinetics-400 HMDB51
Benchmarks: Something-Something V2 Kinetics-400 HMDB51
Metrics: top-1 accuracy top-5 accuracy parameter count training runtime per epoch
Baselines: VideoMAE V2-g TubeViT MVD InternVideo2-1B InternVideo2-6B ILA ATM TAM SlowFast MTV
Builds on: Dynamic Time Warping DBA (DTW Barycenter Averaging) Soft-DTW VideoMAE V2
Closest papers by meaning
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.87)
- Unifying Specialized Visual Encoders for Video Language Models (ICML 2025 · similarity 0.87)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.86)
- A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames (CVPR 2024 · similarity 0.86)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.86)
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.