Research map / Video / Video generation
Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoordTok, a video tokenizer that encodes a long video into factorized triplane latents and decodes by reconstructing randomly sampled space-time patches from (x,y,t) coordinates, enabling training on 128-frame videos.
Problem
Training video tokenizers on long videos is computationally expensive because they reconstruct all frames at once, so existing tokenizers only handle short (about 16-frame) clips and cannot exploit long-range temporal coherence.
Approach
Encode a video into factorized triplane latents (zxy, zyt, zxt) and train the decoder to predict only randomly sampled space-time patches given bilinearly interpolated coordinate-based triplane features, avoiding full-frame reconstruction and making long-video training feasible; a short LPIPS fine-tuning phase on sampled full frames then improves quality.
Key results
- CoordTok encodes a 128-frame 128x128 video into 1280 tokens with rFVD 102.9 and PSNR 28.6, while PVDM-AE needs 1152 tokens for rFVD 1270.1 and baselines need 6144-8192 tokens for comparable quality.
- CoordTok-SiT-L/2 achieves FVD 369.3 on UCF-101 128-frame generation, the best among compared methods (next best PVDM-L at 505.0).
- Random patch sampling of 3.125% of coordinates gives rFVD 238 at 13 max batch size, vs. 479 for random frame sampling at the same ratio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on 128-frame videos and reconstruct randomly sampled coordinate-based patches | CoordTok versus PVDM-AE · UCF-101, 128-frame reconstruction with approximately similar token counts | rFVD: 1270.1 -> 102.9 | Helped |
| use 1280 rather than 3072 tokens (ablation) | SiT-L/2 trained with CoordTok · UCF-101, 128-frame video generation | FVD | Helped |
| increase model size from CoordTok-B to CoordTok-L (ablation) | CoordTok · UCF-101 video reconstruction | PSNR: 25.2 -> 26.9 | Helped |
| increase the spatial dimensions of the triplane representations (ablation) | CoordTok · UCF-101 video reconstruction | reconstruction quality | Helped |
| increase the temporal dimensions of the triplane representations (ablation) | CoordTok · UCF-101 video reconstruction | reconstruction quality | Helped |
| sample patches randomly across the video rather than sample whole frames (ablation) | CoordTok · UCF-101 video reconstruction, 3.125% sampling | rFVD: 479 -> 238 | Helped |
| reduce random patch sampling from 3.125% to 1.563% (ablation) | CoordTok · UCF-101 video reconstruction | rFVD: 238 -> 401 | Hurt |
| reduce random patch sampling from 3.125% to 1.563% (ablation) | CoordTok · UCF-101 tokenizer training | maximum batch size: 13 -> 21 | Helped |
| encode videos into factorized triplane representations rather than 3D latents (ablation) | CoordTok · UCF-101 videos grouped by motion dynamics | correlation between reconstruction quality and dynamics magnitude: -0.40 -> -0.87 | Hurt |
| use coordinate-based representations for patch reconstruction (ablation) | CoordTok · UCF-101 videos grouped by frequency | correlation between reconstruction quality and frequency magnitude: -0.85 -> -0.37 | Helped |
Surprising findings
- Sampling only about 3% of video patches per training step achieves strong reconstruction performance.
- Reconstruction quality of CoordTok correlates strongly and negatively with video dynamics (r = -0.87), indicating difficulty on dynamic videos.
Limitations
- Struggles to reconstruct dynamic videos because content and motion decomposition into triplanes is difficult.
- Evaluated only on UCF-101 at 128x128 resolution.
- Uses a single content plane across time, which may limit representation of dynamics.
Experimental setup
Datasets: UCF-101
Benchmarks: UCF-101
Metrics: rFVD PSNR LPIPS SSIM FVD maximum batch size time (s) memory (GB)
Baselines: MaskGIT-AE TATS-AE MAGVIT-AE-L LARP OmniTokenizer PVDM-AE CosmosTokenizer-CV MoCoGAN StyleGAN-V PVDM-L HVDM Latte-L/2
Builds on: PVDM triplane representations ViT SiT MAGVIT
Closest papers by meaning
- LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior (ICLR 2025 · similarity 0.88)
- OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation (NeurIPS 2024 · similarity 0.88)
- CoANeRV: Coordinate-Aware Token-Space Neural Video Representation (NeurIPS 2026 · similarity 0.88)
- ElasticTok: Adaptive Tokenization for Image and Video (ICLR 2025 · similarity 0.88)
- VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models (NeurIPS 2025 · similarity 0.87)
- End-to-End Vision Tokenizer Tuning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.