Research map / Video / Video segmentation and tracking

TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

NeurIPS 2026 · Video › Video segmentation and tracking · approach: Self-supervised video representation learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TT-VidT, a DINOv3-initialized ViT-B/16 with a compact Temporal Transfer Layer trained with Diff Compression (reconstruct later frames from a first-frame anchor plus motion tokens), and shows in a matched 4x6 architecture-objective sweep that it leads motion-sensitive benchmarks while trailing on appearance-heavy ones.

Problem

Video SSL comparisons confound architecture, objective, data, schedule, scale and decoder, making it unclear which choices yield motion-prioritized representations that improve on frame-to-frame change rather than appearance.

Approach

Pair a compact cross-frame Temporal Transfer pathway (motion tokens, optionally with downsampled spatial tokens) with a Diff Compression objective that reconstructs each target frame from a wide first-frame appearance anchor and a per-frame motion token, so the narrow bottleneck is forced to carry motion information.

Key results

• TT3D with Diff Compression reaches 53.89 on Jester and 18.28 on SSv2 in the no-augmentation sweep, versus 39.47 and 14.95 for ViT3D+MAE (VideoMAE-style).
• Final TT-VidT reports 73.25 on Jester, 25.92 on SSv2, 37.47 on ARID and 18.63 on Diving48 fine-tuning, leading the compared methods on these four benchmarks simultaneously.
• TT-VidT uses 456.1 GFLOPs encoder cost versus 1012.6 for VideoMAE and 874.7 for DisMo at T=8, 256^2; it trails VideoMAE on HMDB51 (25.15 vs 27.73) and DisMo on IARD (74.84 vs 89.89).

Surprising findings

• Larger decoder (L) underperforms the small decoder by roughly 50% on motion-focused benchmarks, suggesting decoder capacity shifts reconstruction burden away from motion tokens.
• TT3D paired with MAE does not reach the strong motion regime; the gain appears only from the pairing with Diff Compression.

Limitations

• Small matched recipe (~1.7M clips, 8 epochs, single-run sweep cells) so conclusions are limited to this regime.
• Does not lead HMDB51, IARD or EPIC-Kitchens verb classification, where appearance-sensitive baselines are stronger.
• Motion-prioritized framing relies on dataset descriptions and a heuristic appearance diagnostic rather than a formal benchmark.

Experimental setup

Datasets: OpenVid Moments-in-Time v2 ImageNet-1k Jester Something-Something V2 ARID Diving48 HMDB51 IARD EPIC-Kitchens-100

Benchmarks: Jester Something-Something V2 ARID Diving48 HMDB51 IARD EPIC-Kitchens-100

Metrics: top-1 accuracy kNN accuracy flip/stay rate under motion inversion FLOPs

Baselines: VideoMAE DisMo V-JEPA 2 ViT3D+MAE MAE-Diff Adaptive AR

Builds on: VideoMAE V-JEPA 2 DisMo VTok DINOv3 DiT

Closest papers by meaning

• Self-Supervised Video Representation Learning by Context and Motion Decoupling (CVPR 2021 · similarity 0.89)
• TimeBridge: Self-Supervised Video Representation Learning via Start-End Joint Embedding and In-Between Frame Prediction (CVPR 2026 · similarity 0.89)
• HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding (NeurIPS 2026 · similarity 0.88)
• LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation (NeurIPS 2026 · similarity 0.87)
• Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.87)
• DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.87)