Research map / Image generation / Autoregressive and latent generation
SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SoftVQ-VAE, a fully differentiable soft-categorical VQ-style continuous 1D tokenizer with representation alignment to DINOv2, compressing ImageNet images to 32-64 tokens and speeding up DiT/SiT/MAR generation.
Problem
Image tokenizers for denoising-based generative models produce too many latent tokens (256-1024), making Transformer generators expensive, and high compression degrades reconstruction and latent semantics in KL-VAE and VQ-VAE.
Approach
Replace VQ-VAE's hard codeword assignment with a softmax over distances to a learnable codebook so each 1D latent token is a soft mixture of codewords; the resulting fully differentiable latent can be directly aligned to pre-trained DINOv2 features via a cosine loss.
Key results
- SiT-XL with SoftVQ-BL/L at 64 tokens reaches gFID 1.78 on ImageNet 256x256 with CFG, and 2.21 at 512x512 with SoftVQ-BL 64 tokens.
- SoftVQ improves inference throughput by up to 18x at 256x256 and 55x at 512x512 relative to KL-tokenizer baselines, and reduces training iterations by 2.3x (3M vs 7M steps).
- SoftVQ-L 32 tokens with DiT-XL gives gFID improvement of 0.62 and SiT-XL 0.71 over KL-tokenizer 1024-token baselines without CFG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SoftVQ-L with 64 tokens instead of a KL tokenizer with 1024 tokens | SiT-XL · ImageNet-1K 256×256 | gFID (w/ CFG): 2.06 -> 1.86 | Helped |
| Use SoftVQ-B with 64 tokens instead of a KL tokenizer with 1024 tokens | SiT-XL · ImageNet-1K 256×256 | gFID (w/ CFG): 2.06 -> 1.78 | Helped |
| Use SoftVQ-BL with 64 tokens instead of a KL tokenizer with 1024 tokens | SiT-XL · ImageNet-1K 512×512 | gFID (w/ CFG): 2.62 -> 2.21 | Helped |
| Use SoftVQ-L with 32 tokens instead of a KL tokenizer with 1024 tokens | SiT-XL · ImageNet-1K 256×256 | gFID (w/o CFG): 8.30 -> 7.59 | Helped |
| Add DINOv2-B feature alignment and encoder initialization (ablation) | SoftVQ-B · ImageNet-1K 256×256 | gFID: 20.33 -> 10.13 | Helped |
| Add feature alignment with DINOv2-B (ablation) | SoftVQ-B initialized with DINOv2-B · ImageNet-1K 256×256 | gFID: 17.20 -> 10.13 | Helped |
| Initialize the encoder with DINOv2-B (ablation) | SoftVQ-B aligned with DINOv2-B · ImageNet-1K 256×256 | gFID: 10.96 -> 10.13 | Helped |
| Align and initialize using CLIP-B instead of DINOv2-B (ablation) | SoftVQ-B · ImageNet-1K 256×256 | gFID: 10.13 -> 9.89 | Helped |
| Align and initialize using EVA-02-B instead of DINOv2-B (ablation) | SoftVQ-B · ImageNet-1K 256×256 | gFID: 10.13 -> 10.34 | No clear effect |
| Reduce SoftVQ-S token count from 256 to 32 (ablation) | SoftVQ-S · ImageNet-1K 256×256 | rFID: 0.80 -> 1.24 | Hurt |
| Reduce VQ-S token count from 256 to 32 (ablation) | VQ-S · ImageNet-1K 256×256 | rFID: 1.45 -> 10.97 | Hurt |
| Reduce AE-S token count from 256 to 32 (ablation) | AE-S · ImageNet-1K 256×256 | rFID: 1.15 -> 2.01 | Hurt |
Surprising findings
- Better tokenizer reconstruction (rFID) does not necessarily translate into better generation gFID; latent-space linear probing accuracy correlates more closely with generation quality.
- Increasing latent dimension improves reconstruction but deteriorates generation performance.
Limitations
- Relies on pre-trained DINOv2 encoder initialization and alignment for best semantic latent space.
- MAR-H results with SoftVQ are slightly worse than KL tokenizer baseline, possibly due to lowered learning rate needed to avoid NaN issues.
- Higher latent dimension hurts generation despite better reconstruction.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet 256x256 ImageNet 512x512
Metrics: rFID gFID Inception Score Precision Recall GFLOPs throughput (imgs/sec) linear probing accuracy
Baselines: DiT-XL/2 SiT-XL/2 SiT-XL + REPA MAR-H TiTok-S-128 TiTok-B DC-AE LlamaGen-3B MaskGIT VQ-S AE-S LDM-4
Builds on: VQ-VAE KL-VAE REPA TiTok LlamaGen MAR SiT
Closest papers by meaning
- Epsilon-VAE: Denoising as Visual Decoding (ICML 2025 · similarity 0.89)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.89)
- Efficient Generative Modeling with Residual Vector Quantization-Based Tokens (ICML 2025 · similarity 0.89)
- End-to-End Training for Unified Tokenization and Latent Denoising (NeurIPS 2026 · similarity 0.89)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.89)
- An Image is Worth 32 Tokens for Reconstruction and Generation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.