AI Paper Atlas

Research map / Video / Video generation

TrackTok: Object-Centric Video Tokenization with Semantically Persistent Tokens

Nhi Pham, Christopher Wewer, Bernt Schiele, Jonas Fischer, Jan Eric Lenssen

NeurIPS 2026 · Video › Video generation · approach: Compositional text-to-video diffusion

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TrackTok, an efficiency-focused video tokenizer that yields object-centric tokens which keep the same semantic identity across time as objects move, deform, or get partially occluded.

Problem

Patch-based and existing holistic video tokenizers tie token identities to fixed image locations, so the same object is not consistently represented across frames.

Approach

Learns temporally persistent object-centric tokens whose identities follow semantic objects through time rather than fixed spatial positions, to improve token efficiency for video.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add cluster queries (CQ) to the encoder (ablation)TrackTok · YouTube-VIS 2021 object discoveryFG-ARI (per frame): 9.9 -> 21.3Improved
add cluster queries (CQ) to the encoder (ablation)TrackTok · YouTube-VIS 2021 object trackingJ&F: 10.5 -> 19.1Improved
add cluster queries (CQ) to the encoder (ablation)TrackTok · YouTube-VIS 2021 token puritytoken purity: 84.9 -> 93.8Improved
add cluster queries (CQ) to the encoder (ablation)TrackTok · UCF-101 class-conditional generation, all token budgetsgFVDImproved
use V-JEPA 2.1 features plus cluster queries instead of continuous LARP-L (no VFM, no CQ) (ablation)TrackTok · UCF-101 generationgFVDImproved
use V-JEPA 2.1 features as encoder input (ablation)TrackTok · UCF-101 reconstructionrFVDHurt
replace LARP with TrackTok at 32 tokensvideo tokenizer · UCF-101 class-conditional generation (32 tokens)gFVD: 672 -> 201Improved
replace LARP with TrackTok at 1024 tokensvideo tokenizer · UCF-101 class-conditional generation (1024 tokens)gFVD: 102 -> 117Hurt
use TrackTok at 128 tokens instead of 1024-token LARPvideo tokenizer · UCF-101 class-conditional generationgFVD: 107 -> 126No clear effect
use TrackTok instead of LARP and AdapTok at equal low token countsvideo tokenizer · Kinetics-600 frame predictiongFVD: 4.9x and 10.8x better gFVDImproved
use TrackTok at 512 tokens and above for reconstructionvideo tokenizer · UCF-101 reconstructionrFVDHurt
use TrackTok instead of LARP at 64 tokens for reconstructionvideo tokenizer · UCF-101 reconstruction (64 tokens)rFVD: 5.2x lower rFVD than LARPImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy