Research map / Video / Video-language understanding
ElasticTok: Adaptive Tokenization for Image and Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ElasticTok, an autoencoder tokenizer trained with random end-of-sequence token masking so a frame or video block can be encoded into a variable number of tokens conditioned on prior frames.
Problem
Video tokenizers encode each frame or block into a fixed number of tokens, wasting tokens on simple content and losing information on complex content, which makes long video modeling computationally expensive.
Approach
Sample a random token length per block during training and mask the tail of the encoder output (left-aligned tokens), so a single autoencoder learns elastic encodings; at inference, search for the shortest length meeting a reconstruction threshold, with block-causal conditioning on previous frames.
Key results
- ElasticTok matches fixed-token baselines' reconstruction satisfaction rates while using 1.3x-5x fewer tokens depending on threshold.
- On videos at the 0.003 MSE threshold with 50% average tokens, ElasticTok-VAE satisfies 88% of blocks versus 78% for the 50% fixed baseline.
- Binary search achieves 7% token-count error with 12 NFEs; neural regression achieves 9% error with 1 NFE; full search uses 4096 NFEs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive token lengths instead of fixed 50% token usage | ElasticTok-VAE · Video blocks, MSE threshold 0.001 | blocks satisfying reconstruction threshold: 40% -> 50% | Helped |
| Use adaptive token lengths instead of fixed 50% token usage | ElasticTok-VAE · Video blocks, MSE threshold 0.003 | blocks satisfying reconstruction threshold: 78% -> 88% | Helped |
| Use adaptive token lengths instead of fixed 50% token usage | ElasticTok-VAE · Video blocks, MSE threshold 0.015 | blocks satisfying reconstruction threshold: 99% -> 99% | No clear effect |
| Use adaptive token lengths rather than fixed token allocation at matched worst-case reconstruction | ElasticTok-FSQ · Video, worst-case MSE threshold 0.001 | average percentage of tokens used: 78% -> 75% | Helped |
| Use adaptive token lengths rather than fixed token allocation at matched worst-case reconstruction | ElasticTok-FSQ · Video, worst-case MSE thresholds 0.003 and 0.015 | average percentage of tokens used: 57% -> 37%; 15% -> 9.4% | Helped |
| Replace the fixed-token VLM tokenizer with adaptive ElasticTok tokens | OpenLLaMA-3B VLM · GQA and POPE | accuracy: GQA: 54% -> 54%; POPE: 82% -> 82% | No clear effect |
| Replace the fixed-token VLM tokenizer with adaptive ElasticTok tokens | OpenLLaMA-3B VLM · MSVD-QA and MSRVTT-QA | accuracy: MSVD: 53% -> 52%; MSRVTT: 37% -> 37% | Mixed |
| Use encoder conditioning on the token mask (ablation) | ElasticTok-FSQ · Video blocks, MSE threshold 0.001 | blocks satisfying reconstruction threshold: 33.1% -> 36% | Helped |
| Use encoder conditioning on the token mask (ablation) | ElasticTok-FSQ · Video blocks, MSE threshold 0.003 | blocks satisfying reconstruction threshold: 75.3% -> 79.6% | Helped |
| Use encoder conditioning on the token mask (ablation) | ElasticTok-FSQ · Video blocks, MSE threshold 0.015 | blocks satisfying reconstruction threshold: 98.4% -> 98.5% | No clear effect |
| Use Neural Regression instead of Full Search | ElasticTok-FSQ inference · ElasticTok-FSQ inference | NFEs: 4096 -> 1 | Helped |
| Use Neural Regression instead of Full Search | ElasticTok-FSQ inference · ElasticTok-FSQ inference | error rate: 0% -> 9% | Hurt |
Surprising findings
- Performance degrades at the maximum context length (1M tokens), attributed to limited training compute at that length.
- Dropping the encoder conditioning on the mask yields only slightly worse reconstruction while allowing roughly 2x faster inference.
Limitations
- Slightly underperforms fixed-token baselines at the tail ends of encoding lengths; left-aligned masking may cause conflicting representations.
- No GAN is used, so reconstructions may lack sharpness; training was not stable with GAN on very long sequences.
- Performance degrades at the maximum 1M-token context due to limited training compute.
Experimental setup
Datasets: COYO-700M ImageNet WebVid10M Video-ChatGPT instruct data custom 6M video dataset from Common Crawl
Benchmarks: GQA POPE MSVD-QA MSRVTT-QA
Metrics: reconstruction MSE threshold satisfaction rate token usage VQA accuracy error rate NFEs Pearson correlation
Baselines: fixed-token autoencoders at varying token capacities OpenLLaMA-3B with fixed 50% token baseline
Builds on: FSQ VAE RingAttention LWM nested dropout
Closest papers by meaning
- FlexTok: Resampling Images into 1D Token Sequences of Flexible Length (ICML 2025 · similarity 0.88)
- Adaptive Length Image Tokenization via Recurrent Allocation (ICLR 2025 · similarity 0.88)
- Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction (CVPR 2025 · similarity 0.88)
- VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models (NeurIPS 2025 · similarity 0.87)
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models (CVPR 2025 · similarity 0.86)
- Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.