Research map / Video / Video-language understanding
One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory
Chenhao Zheng, Jieyu Zhang, Mohammadreza Salehi, Ziqi Gao, Vishnu Iyengar, Norimasa Kobori, Quan Kong, Ranjay Krishna
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| tokenize videos by panoptic sub-object trajectories instead of space-time patches | video encoder (ViT-Large) · zero-shot video-text retrieval | R@5: ActivityNet txt2vid 35.3 -> 38.6 | Improved |
| use trajectory tokens instead of space-time patches on SSV2 | TrajViT · SSV2 action classification | top-1 accuracy: 46.1 -> 45.8 | No clear effect |
| use TrajViT as VideoLLM encoder instead of ViT3D | Llama3 VideoLLM · NextQA OE | WUPS: 49.2 -> 65.1 | Improved |
| use TrajViT as VideoLLM encoder instead of ViT3D | Llama3 VideoLLM · MLVU | accuracy: 19.2 -> 32.2 | Improved |
| add segmentation tracking to segmentation-only tokens (ablation) | TrajViT · ablation 880k, K400 | K400 accuracy: 19.9 -> 25.9 | Improved |
| add segmentation to tracking-only tokens (ablation) | TrajViT · ablation 880k, retrieval | retrieval: 15.4 -> 17.8 | Improved |
| use DirectSAM instead of SAM for segmentation (ablation) | TrajViT · ablation 880k, K400 | K400 accuracy: 23.5 -> 25.9 | Improved |
| use 1024 instead of 512 trajectory resolution (ablation) | TrajViT · ablation 880k | SSV2/K400/Retrieval: SSV2 22.5 vs 22.7; K400 26.3 vs 25.9 | No clear effect |
| use ResNet-18 instead of Hiera-small as feature extractor (ablation) | TrajViT trajectory encoder · ablation 880k, K400 | K400 accuracy: 21 -> 25.9 | Improved |
| add explicit position branch (ablation) | TrajViT trajectory encoder · ablation 880k, SSV2 | SSV2 accuracy: 20.6 -> 22.7 | Improved |
| use perceiver resampler instead of mean-pooling to aggregate frames (ablation) | TrajViT trajectory encoder · ablation 880k, SSV2 | SSV2 accuracy: 19.2 -> 22.7 | Improved |
| increase query tokens per trajectory from 1 to 4 (ablation) | TrajViT trajectory encoder · ablation 880k, K400 | K400 accuracy: 25.9 -> 26.2 | No clear effect |
Closest papers by meaning
- Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction (CVPR 2025 · similarity 0.89)
- TrajLoc: Trajectory-Attention Localization for Multi-Object Motion Control (NeurIPS 2026 · similarity 0.88)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.88)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.87)
- V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models (NeurIPS 2026 · similarity 0.87)
- Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
