Research map / Video / Action recognition

TrAction: Action Recognition with Sparse Trajectories

NeurIPS 2026 · Video › Action recognition · approach: Disentangled action recognition · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TrAction, a small transformer operating only on 2.5D point trajectories (CoTracker3 + Video Depth Anything) with masked-trajectory pretraining, showing trajectories reach 45.2% SSv2 and 54.1% EK100 and are complementary to DINOv2 and V-JEPA 2 under late fusion.

Problem

Action recognition models rely on dense RGB volumes that are costly and prone to appearance and background shortcuts; it is unclear how much action information sparse point trajectories carry and whether they complement appearance-based vision encoders.

Approach

Treat each 2.5D point trajectory (2D track plus monocular depth and visibility) as a token via a per-trajectory MLP, process sets of trajectories with a small transformer, pretrain with masked trajectory reconstruction, and late-fuse with frozen appearance encoders.

Key results

• Trajectory-only model reaches 45.2% top-1 on SSv2 and 54.1% on EK100 with under 8M parameters, outperforming flow-only I3D on SSv2 (38.4%).
• Late fusion with DINOv2-B (16 frames) improves SSv2 top-1 from 53.6 to 62.3 (+8.7); fusion with V-JEPA 2-L improves SSv2 by +1.6 to 68.3.
• On Chirality-in-Action (SSv2 split) the trajectory model scores 86.2%, above VideoMAE-L (85.7) and V-JEPA-L (85.4); masked pretraining adds about 4.9 (SSv2) and 5.1 (EK100) points.

Surprising findings

• Trajectory-only model is competitive with or better than dense vision baselines on time-reversal sensitivity (Chirality-in-Action), surpassing V-JEPA-L.
• Trajectories with 2D only (no depth) drop SSv2 accuracy by about 9.5 points, showing depth is substantial for this modality.

Limitations

• Depends on a separate, computationally expensive trajectory extraction step; trackers struggle with rapid camera motion or abrupt scene changes.
• Trajectories are uninformative on appearance-dominated benchmarks such as Kinetics-400, UCF-101, and HMDB-51.

Experimental setup

Datasets: Kinetics-400 Something-Something V2 EPIC-Kitchens-100 IARD Chirality-in-Action

Benchmarks: Something-Something V2 EPIC-Kitchens-100 Kinetics-400 IARD Chirality-in-Action

Metrics: top-1 accuracy kNN accuracy identity accuracy

Baselines: I3D (flow-only) DINOv2-B SigLIP-2-B VideoPrism-B V-JEPA 2-L VideoMAE-L DisMo-B

Builds on: CoTracker3 Video Depth Anything masked autoencoder V-JEPA

Closest papers by meaning

• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (NeurIPS 2021 · similarity 0.87)
• MARS: Motion-Augmented RGB Stream for Action Recognition (CVPR 2019 · similarity 0.87)
• Video Action Transformer Network (CVPR 2019 · similarity 0.86)
• VA-AR: Learning Velocity-Aware Action Representations with Mixture of Window Attention (AAAI 2025 · similarity 0.85)
• DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.85)
• FROSTER: Frozen CLIP is A Strong Teacher for Open-Vocabulary Action Recognition (ICLR 2024 · similarity 0.85)