Research map / Video / Video generation
TrackTok: Object-Centric Video Tokenization with Semantically Persistent Tokens
Nhi Pham, Christopher Wewer, Bernt Schiele, Jonas Fischer, Jan Eric Lenssen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TrackTok, an efficiency-focused video tokenizer that yields object-centric tokens which keep the same semantic identity across time as objects move, deform, or get partially occluded.
Problem
Patch-based and existing holistic video tokenizers tie token identities to fixed image locations, so the same object is not consistently represented across frames.
Approach
Learns temporally persistent object-centric tokens whose identities follow semantic objects through time rather than fixed spatial positions, to improve token efficiency for video.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cluster queries (CQ) to the encoder (ablation) | TrackTok · YouTube-VIS 2021 object discovery | FG-ARI (per frame): 9.9 -> 21.3 | Improved |
| add cluster queries (CQ) to the encoder (ablation) | TrackTok · YouTube-VIS 2021 object tracking | J&F: 10.5 -> 19.1 | Improved |
| add cluster queries (CQ) to the encoder (ablation) | TrackTok · YouTube-VIS 2021 token purity | token purity: 84.9 -> 93.8 | Improved |
| add cluster queries (CQ) to the encoder (ablation) | TrackTok · UCF-101 class-conditional generation, all token budgets | gFVD | Improved |
| use V-JEPA 2.1 features plus cluster queries instead of continuous LARP-L (no VFM, no CQ) (ablation) | TrackTok · UCF-101 generation | gFVD | Improved |
| use V-JEPA 2.1 features as encoder input (ablation) | TrackTok · UCF-101 reconstruction | rFVD | Hurt |
| replace LARP with TrackTok at 32 tokens | video tokenizer · UCF-101 class-conditional generation (32 tokens) | gFVD: 672 -> 201 | Improved |
| replace LARP with TrackTok at 1024 tokens | video tokenizer · UCF-101 class-conditional generation (1024 tokens) | gFVD: 102 -> 117 | Hurt |
| use TrackTok at 128 tokens instead of 1024-token LARP | video tokenizer · UCF-101 class-conditional generation | gFVD: 107 -> 126 | No clear effect |
| use TrackTok instead of LARP and AdapTok at equal low token counts | video tokenizer · Kinetics-600 frame prediction | gFVD: 4.9x and 10.8x better gFVD | Improved |
| use TrackTok at 512 tokens and above for reconstruction | video tokenizer · UCF-101 reconstruction | rFVD | Hurt |
| use TrackTok instead of LARP at 64 tokens for reconstruction | video tokenizer · UCF-101 reconstruction (64 tokens) | rFVD: 5.2x lower rFVD than LARP | Improved |
Closest papers by meaning
- Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction (CVPR 2025 · similarity 0.90)
- LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior (ICLR 2025 · similarity 0.89)
- Language Model Beats Diffusion - Tokenizer is key to visual generation (ICLR 2024 · similarity 0.89)
- CORE: Compact Object-centric REpresentations as a New Paradigm for Token Merging in LVLMs (CVPR 2026 · similarity 0.88)
- OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation (NeurIPS 2024 · similarity 0.88)
- ElasticTok: Adaptive Tokenization for Image and Video (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
