Research map / Image generation / Autoregressive and latent generation
Language-Guided Image Tokenization for Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TexTok, which conditions a 1D image tokenizer and detokenizer on T5 embeddings of captions so that fewer tokens can encode finer visual detail, and uses a DiT generator on the resulting tokens.
Problem
Image tokenizers face a trade-off between compression rate and reconstruction quality, which makes high-resolution latent generation expensive.
Approach
Inject text caption embeddings (frozen T5) into both the tokenizer and detokenizer of a 1D ViT tokenizer so high-level semantics come from language and learnable tokens can focus on fine visual detail, improving reconstruction and generation at fewer tokens.
Key results
- TexTok improves average reconstruction FID by 29.2% (ImageNet-256) and 48.1% (ImageNet-512) over the no-text baseline across token counts.
- TexTok-256 + DiT-XL reaches gFID 1.46 on ImageNet 256x256 and 1.62 on ImageNet 512x512 without classifier-free guidance caveats, state of the art per the paper.
- Replacing the DiT tokenizer with TexTok-32 gives a 93.5x inference speedup on ImageNet 512x512 while still outperforming DiT-XL/2 gFID.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add text conditioning to the tokenizer and detokenizer | TexTok · ImageNet 256×256, 32 tokens | rFID: 3.82 -> 2.40 | Helped |
| Add text conditioning to the tokenizer and detokenizer | TexTok · ImageNet 512×512, 32 tokens | gFID: 9.22 -> 3.61 | Helped |
| Use TexTok-256 instead of the compared tokenizer | DiT-XL · ImageNet 256×256 class-conditional generation | FID: 1.58 -> 1.46 | Helped |
| Use TexTok-256 instead of the compared tokenizer | DiT-XL · ImageNet 512×512 class-conditional generation | FID: 1.81 -> 1.62 | Helped |
| Add text conditioning to the tokenizer and detokenizer | DiT-XL-T2I · ImageNet 256×256 text-to-image, 128 tokens | FID: 3.01 -> 2.82 | Helped |
| Use 75-word captions instead of no text conditioning (ablation) | TexTok-Base-128 · ImageNet 256×256 reconstruction | rFID: 1.49 -> 1.04 | Helped |
| Use T5-XXL instead of T5-XL (ablation) | TexTok-Base-128 · ImageNet 256×256 reconstruction | rFID: 1.04 -> 0.99 | Helped |
| Use in-context conditioning instead of a cross-attention layer (ablation) | TexTok-Base-128 · ImageNet 256×256 reconstruction | rFID: 1.31 -> 1.04 | Helped |
| Apply text conditioning to tokenizer and detokenizer rather than tokenizer only (ablation) | TexTok-Base-128 · ImageNet 256×256 reconstruction | rFID: 1.38 -> 1.04 | Helped |
| Increase model size from TexTok-Small to TexTok-Base (ablation) | TexTok · ImageNet 256×256 reconstruction | rFID: 1.35 -> 1.04 | Helped |
| Increase model size from TexTok-Base to TexTok-Large (ablation) | TexTok · ImageNet 256×256 reconstruction | rFID: 1.04 -> 1.03 | Helped |
Surprising findings
- The no-text baseline with 8x fewer tokens already outperforms SD-VAE-f8 in both reconstruction and generation on ImageNet 256x256.
- Larger T5 encoder and richer captions help, but increasing tokenizer size beyond TexTok-Base gives only marginal gains.
Limitations
- Requires captions for each training image; ImageNet captions were produced by a VLM (Gemini 1.5 Flash) offline.
- Evaluated primarily on ImageNet; text-to-image results are limited to ImageNet captions.
- Class-conditional generation uses pre-generated captions at inference rather than captions from the generator.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 ImageNet 512x512
Metrics: rFID rIS PSNR SSIM LPIPS gFID gIS precision recall FID
Baselines: SD-VAE-f8 TiTok-L-32 TiTok-64 TiTok-128 MAGVIT-v2 MaskBit MAR VAR DiT-XL/2 LlamaGen-3B Baseline (w/o text)
Builds on: TiTok DiT MAGVIT-v2 LlamaGen
Closest papers by meaning
- An Image is Worth 32 Tokens for Reconstruction and Generation (NeurIPS 2024 · similarity 0.90)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.88)
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.88)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.88)
- FlexTok: Resampling Images into 1D Token Sequences of Flexible Length (ICML 2025 · similarity 0.88)
- SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.