Research map / Video / Video generation
Image and Video Tokenization with Binary Spherical Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BSQ-ViT, a transformer image/video tokenizer that projects latents onto a hypersphere, binarizes them (Binary Spherical Quantization), and uses block-wise causal attention, reaching state-of-the-art image/video reconstruction and competitive video compression.
Problem
Visual tokenizers based on CNNs and vector quantization scale poorly with codebook size, are costly for video, and do not handle variable-length videos or exploit global structure for compression.
Approach
Replace CNN and VQ bottlenecks with a ViT encoder/decoder and a binary quantizer applied to l2-normalized low-dimensional projections, giving an implicit codebook of size 2^L with no learned codebook; soft quantization factorizes per dimension so entropy regularization is linear in L; block-wise causal masking unifies images and videos.
Key results
- On ImageNet-1k 256x256 reconstruction, the 36-bit BSQ-ViT reaches rFID 0.41 versus 0.72 for SDXL-VAE, at 2.4x its throughput.
- On UCF-101 video reconstruction, the causal 36-bit BSQ model reduces rFVD from 8.62 (MAGVIT-v2) to 4.10 on the train split.
- Masked LM with BSQ achieves FID 5.44 on ImageNet 128x128 generation, versus 6.02 for BigGAN and 5.91 for ADM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the image tokenizer on video data | BSQ-ViT, 18 bits · UCF-101 val, split 1 | rFVD: 342 to 11.62 | Helped |
| Increase BSQ bottleneck from 18 to 36 bits (ablation) | Causal BSQ-ViT · UCF-101 val, split 1 | rFVD: 11.62 to 6.21 | Helped |
| Use non-blockwise-causal attention instead of blockwise-causal attention (ablation) | BSQ-ViT · UCF-101 video reconstruction | rFVD: 8.08 to 9.16 | Helped |
| Increase BSQ bottleneck from 10 to 18 bits (ablation) | BSQ tokenizer · ImageNet-1k val, 128×128 | rFID: 4.51 to 2.66 | Helped |
| Increase VQ codebook from 16384 to 65536 entries (ablation) | VQ tokenizer · ImageNet-1k val, 128×128 | rFID: 4.27 to 6.61 | Hurt |
| Remove ℓ2 normalization from BSQ (ablation) | BSQ tokenizer · ImageNet-1k val, 128×128 | rFID: 2.66 to 30.7 | Hurt |
| Remove commitment loss (ablation) | BSQ tokenizer · ImageNet-1k val, 128×128 | rFID: 2.95 to 2.83 | Helped |
| Remove conditional entropy term H(p(c|u)) (ablation) | BSQ tokenizer · ImageNet-1k val, 128×128 | rFID: 2.95 to 2.44 | Helped |
| Remove dataset entropy maximization term (ablation) | BSQ tokenizer · ImageNet-1k val, 128×128 | rFID: 2.95 to 13.8 | Hurt |
| Remove perceptual loss (ablation) | BSQ tokenizer · ImageNet-1k val, 128×128 | rFID: 2.95 to 19.2 | Hurt |
| Use factorized dataset entropy approximation instead of group size 9 (ablation) | BSQ tokenizer · ImageNet-1k val, 128×128 | rFID: 2.83 to 2.86 | No clear effect |
| Use factorized dataset entropy approximation rather than enumerating full codebook entropy (ablation) | BSQ tokenizer · ImageNet-1k val, 128×128 | speed: 0.212 ms | Helped |
Surprising findings
- Removing the l2 normalization (equivalent to LFQ) drops code usage to 0.6% and worsens rFID to 30.7 with a ViT encoder, indicating LFQ struggles with ViT tokenizers.
- Non-causal (full attention) video tokenization slightly outperforms block-wise causal attention on reconstruction metrics.
Limitations
- Tested only at 256x256 or 128x128 resolution; higher resolution and variable aspect ratios unexplored.
- Training data limited to ImageNet-1k and UCF-101; video compression lags HEVC and VCT on UVG 1080P.
- Video compression is limited by training on low-resolution MPEG-4 encoded UCF-101 clips.
Experimental setup
Datasets: ImageNet-1k UCF-101 COCO 2017val MCL-JCV UVG
Benchmarks: COCO 2017val ImageNet-1k UCF-101 MCL-JCV UVG
Metrics: PSNR SSIM LPIPS rFID rFVD FID Inception Score precision/recall bpp vs MS-SSIM throughput
Baselines: MaskGIT ViT-VQGAN SD-VAE 1.x SD-VAE 2.x SDXL-VAE MAGVIT-v2 MAGVIT TATS VCT H.264 HEVC BigGAN
Builds on: VQ-VAE ViT-VQGAN LFQ MaskGIT VQGAN
Closest papers by meaning
- Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compression (NeurIPS 2025 · similarity 0.87)
- An Image is Worth 32 Tokens for Reconstruction and Generation (NeurIPS 2024 · similarity 0.86)
- Finite Scalar Quantization: VQ-VAE Made Simple (ICLR 2024 · similarity 0.86)
- OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation (NeurIPS 2024 · similarity 0.86)
- ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation (ICLR 2025 · similarity 0.86)
- Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.