Research map / Image generation / Autoregressive and latent generation
Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales a ViT-based (Llama-upgraded, asymmetric encoder-decoder) visual tokenizer ViTok for image and video reconstruction and generation, finding bottleneck size E dominates reconstruction while encoder/decoder scaling gives limited generation gains.
Problem
Visual tokenizers (auto-encoders) for latent diffusion are almost always CNN-based and rarely scaled; it is unclear how auto-encoder design and scale (bottleneck, encoder, decoder) affect reconstruction and downstream generation.
Approach
Replace CNN tokenizers with an asymmetric ViT auto-encoder upgraded with Llama components (SwiGLU, 3D axial RoPE), train it on large in-the-wild image and video data, and systematically sweep bottleneck, encoder and decoder sizes plus loss choices to identify what drives reconstruction and generation.
Key results
- On 256p ImageNet-1K reconstruction, ViTok S-B/16 reports rFID 0.50 at 34.8 GFLOPs versus SD-VAE 0.78 at 162.2 GFLOPs, with 2-5x fewer FLOPs.
- On 16-frame 128p UCF-101 video, ViTok S-B/4x8 reaches rFVD 8.04 versus MAGViTv2 16.12 and LARP 20.
- Class-conditional 256p ImageNet generation with DiT-XL on ViTok S-B/16 gives gFID 2.45 versus 2.27 for SD-VAE + DiT-XL, i.e. competitive but not better.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase bottleneck E (latent floats) at fixed patch/FLOPs (ablation) | ViTok S-B tokenizer · ImageNet-1K 256p reconstruction | rFID, rPSNR, rSSIM: strong log(E) correlation with reconstruction quality | Helped |
| increase bottleneck E via large channel size c (ablation) | ViTok S-B tokenizer as DiT latent · ImageNet-1K 256p class-conditional generation | gFID, gIS | Hurt |
| scale encoder size (S to B/L) (ablation) | ViTok tokenizer · ImageNet-1K 256p reconstruction | rFID, rPSNR | No clear effect |
| scale encoder size (ablation) | ViTok tokenizer as DiT latent · ImageNet-1K 256p class-conditional generation | gFID, gIS | Hurt |
| scale decoder size (Base to Large) (ablation) | ViTok S-B/16 tokenizer · ImageNet-1K 256p reconstruction | rFID: rFID 1.6 -> 1.3 at E=4096 | Helped |
| scale decoder size (ablation) | ViTok tokenizer as DiT latent · ImageNet-1K 256p class-conditional generation | gFID, gIS | No clear effect |
| add GAN loss with decoder-only fine-tuning (stage 2) (ablation) | ViTok S-B/16 · ImageNet-1K 256p reconstruction | rFID: rFID 0.50 with GAN; lower SSIM/PSNR | Helped |
| add GAN decoder fine-tuning (stage 2) (ablation) | ViTok S-B/16 as DiT latent · ImageNet-1K 256p class-conditional generation (CFG 1.5) | gFID: gFID 5.5 -> 4.9 | Helped |
| increase LPIPS weight in stage 1 losses (ablation) | ViTok S-B/16 · ImageNet-1K 256p reconstruction | rSSIM/rPSNR vs rFID/rIS | Mixed |
| increase frame count (16 to 64) for fixed tubelet size (ablation) | ViTok S-B/4x16 video tokenizer · UCF-101 256p video reconstruction | rFVD, rFID, rPSNR | Helped |
| use larger patch/bottleneck from image E to video (temporal stride and spatial patch changes at fixed E) (ablation) | ViTok S-B video tokenizer · 256p video reconstruction | rFID, rSSIM, rPSNR | No clear effect |
| switch from simple patch tokens to latent-token (1D) variant (ablation) | ViTok S-B/16 · ImageNet-1K 256p reconstruction | rFID, rIS | Hurt |
Surprising findings
- Scaling the encoder does not improve reconstruction and tends to slightly hurt generation.
- Larger bottleneck E improves reconstruction but a larger channel size c degrades generation, so generation performance is second-order in E rather than monotonic.
Limitations
- Generation performance does not beat SD-VAE+DiT on ImageNet 256p; tokenizer scaling gives no generation gain.
- Video experiments limited to 16-frame 128p/256p and UCF-101 class-conditional generation.
- Uses a single DiT generator and a small step budget relative to some baselines.
Experimental setup
Datasets: Shutterstock image dataset ImageNet-1K COCO-2017 Shutterstock video dataset UCF-101 Kinetics-700
Benchmarks: ImageNet-1K COCO-2017 UCF-101 Kinetics-700
Metrics: rFID rIS rSSIM rPSNR gFID gIS rFVD gFVD FVD
Baselines: SD-VAE SDXL-VAE Cosmos-CI OAI TiTok-B TATS MAGViT MAGViTv2 LARP-L-Long W.A.L.T
Builds on: Vision Transformer VAE Llama ViViT DiT
Closest papers by meaning
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.92)
- Epsilon-VAE: Denoising as Visual Decoding (ICML 2025 · similarity 0.90)
- OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation (NeurIPS 2024 · similarity 0.89)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.89)
- ImageFolder: Autoregressive Image Generation with Folded Tokens (ICLR 2025 · similarity 0.88)
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.