Research map / Video / Action recognition
Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LiFT, a self-supervised autoencoder over frozen DINOv2 per-frame CLS features that enforces linear latent trajectories, and a chiral action recognition benchmark (CiA) mined from SSv2, EPIC-Kitchens and Charades to test time-sensitivity.
Problem
Video representations are often insensitive to temporal visual change; existing action benchmarks are dominated by static cues, so there is no clear test of whether descriptors distinguish temporally opposite actions (e.g., opening vs closing a door).
Approach
Train a small transformer autoencoder over frozen DINOv2 frame features that reconstructs the trajectory from two latent tokens (static and dynamic) under a linear-in-time latent constraint (perceptual straightening), with an orthogonality regularizer; the concatenated tokens form a compact time-aware video descriptor. Evaluate via linear probes on mined chiral action pairs.
Key results
- LiFT (768-dim descriptor) achieves 86.6% on CiA SSv2, 75.5% on EPIC and 69.5% on Charades chiral accuracy, beating VideoMAE (ViT-L, time concat., 8192-dim) at 85.7/75.0/66.1.
- Concatenating LiFT with VideoJEPA improves linear-probe accuracy on K400 from 59.8 to 63.7 and on SSv2 from 49.6 to 52.3.
- LiFT underperforms video models on horizontal spatial position changes (75.2 vs 96.3 for VideoMAE), attributed to DINOv2 features encoding horizontal motion weakly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LiFT rather than the strongest listed baseline | chiral action recognition · SSv2 (Chiral) | accuracy: 85.7 -> 86.6 | Helped |
| use LiFT rather than the strongest listed baseline | chiral action recognition · EPIC (Chiral) | accuracy: 75.0 -> 75.5 | Helped |
| use LiFT rather than the strongest listed baseline | chiral action recognition · Charades (Chiral) | accuracy: 67.8 -> 69.5 | Helped |
| combine LiFT with VideoJEPA | VideoJEPA linear probe · Kinetics-400, UCF-101, HMDB-51, SSv2 | accuracy: +3.9; +1.3; +1.9; +2.7 | Helped |
| combine LiFT with VideoMAE | VideoMAE linear probe · Kinetics-400, UCF-101, HMDB-51, SSv2 | accuracy: +8.6; +5.2; +6.1; +6.0 | Helped |
| combine LiFT with InternVid2.5 | InternVid2.5 linear probe · Kinetics-400, UCF-101, HMDB-51, SSv2 | accuracy: +3.1; +2.1; +3.4; +11.5 | Helped |
| replace DINOv2 with language-supervised image encoders (ablation) | LiFT · SSv2 (Chiral) | accuracy: 86.6 -> 75.9; 77.8; 75.7 | Hurt |
| replace DINOv2 with iBOT (ablation) | LiFT · SSv2 (Chiral) | accuracy: 86.6 -> 80.3 | Hurt |
| use larger DINOv2 encoders (ablation) | LiFT · SSv2 (Chiral) | accuracy: 85.4; 85.9 vs 86.6 | No clear effect |
| add orthogonality loss (ablation) | LiFT · SSv2 (Chiral) | accuracy: 85.9 -> 86.6 | Helped |
| vary latent dimension (best at 384; 512 underperforms) (ablation) | LiFT · SSv2 (Chiral) | accuracy: 85.9; 85.8; 86.6; 84.9 | Mixed |
| reduce training data from 100% to 10% (ablation) | LiFT trained on Kinetics-400 · SSv2 (Chiral) | accuracy: 86.2 ± 0.4 -> 83.3 ± 0.1 | Hurt |
Surprising findings
- Naively concatenated DINOv2 image features match or beat much larger video models on chiral action recognition.
- Using only 10% of Kinetics-400 training data yields 83.3% on the SSv2 chiral subset, versus 86.2% with full data.
Limitations
- Uses only per-frame CLS tokens, missing spatial details, notably horizontal translation.
- Weak on horizontal spatial position changes (75.2% vs 96.3% for VideoMAE on that change type).
- Chiral pair construction relies on ChatGPT antonym proposals with manual verification.
Experimental setup
Datasets: Kinetics-400 Something-Something v2 EPIC-Kitchens Charades
Benchmarks: Chirality in Action (CiA) SSv2 CiA EPIC CiA Charades Kinetics-400 UCF-101 HMDB-51 Something-Something v2
Metrics: accuracy
Baselines: CLIP BLIP2 SigLIP SigLIP 2 DINOv2 ST-Adapter DiST Tubelet Contrast TCLR CTP VideoMAE VideoMAEv2
Builds on: DINOv2 Perceptual Straightening hypothesis VideoMAE VideoJEPA
Closest papers by meaning
- TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognition (CVPR 2023 · similarity 0.87)
- Leveraging Temporal Context in Low Representational Power Regimes (CVPR 2023 · similarity 0.86)
- FlexiVideo: Variation-Aware Temporal Dynamics Modeling for Efficient Video Understanding (CVPR 2026 · similarity 0.86)
- Learning by Aligning Videos in Time (CVPR 2021 · similarity 0.86)
- Action Recognition From Single Timestamp Supervision in Untrimmed Videos (CVPR 2019 · similarity 0.86)
- Learning predictable and robust neural representations by straightening image sequences (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.