Research map / Image generation / Autoregressive and latent generation
CAT: Content-Adaptive Image Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAT, a content-adaptive image tokenizer that uses LLM-scored caption complexity to pick 8x/16x/32x compression per image via a nested VAE, and uses variable-length latents for DiT ImageNet generation.
Problem
Fixed-ratio image tokenizers spend the same number of tokens on every image regardless of complexity, wasting compute on simple images and losing detail on complex ones; existing adaptive methods need image input, which is unavailable for text-to-image generation.
Approach
Predict per-image compression ratio from text alone (caption plus face/text yes-no answers scored 1-9 by Llama 3 70B), and train a nested VAE whose intermediate encoder/decoder features are routed to a shared middle block so one model supports multiple latent sizes.
Key results
- DiT-XL with CAT achieves FID 4.56 on 512px ImageNet class-conditional generation versus 4.78 for fixed 16x and 10.03 for LDM VAE, at 0.82x relative FLOPs.
- CAT reduces rFID by 12% on CelebA (1.97 vs 2.25) and 39% on ChartQA (5.27 vs 8.67) relative to fixed 16x.
- CAT improves inference throughput by 18.5% due to an average of 216 tokens at inference versus 256 for fixed tokenization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CAT instead of Fixed 16x | image reconstruction · CelebA | rFID: 2.25 -> 1.97 | Helped |
| use CAT instead of Fixed 16x | image reconstruction · ChartQA | rFID: 8.67 -> 5.27 | Helped |
| use CAT instead of Fixed 16x | image reconstruction · ImageNet | rFID: 0.38 -> 0.46 | Hurt |
| use CAT instead of Fixed 16x | DiT-XL/2 ImageNet generation · ImageNet class-conditional generation | FID: 4.78 -> 4.56 | Helped |
| use CAT instead of Fixed 16x | DiT-XL/2 ImageNet generation · ImageNet class-conditional generation | Eval rFLOPs: 1× -> 0.82× | Helped |
| use thresholds (4, 7) instead of (2, 8), producing a more diverse compression-ratio distribution (ablation) | CAT · ImageNet reconstruction | rFID: 0.43 -> 0.46 | Helped |
| use thresholds (4, 7) instead of (2, 8), producing a more diverse compression-ratio distribution (ablation) | CAT · CelebA reconstruction | rFID: 2.58 -> 1.97 | Helped |
| use thresholds (4, 7) instead of (2, 8), producing a more diverse compression-ratio distribution (ablation) | CAT · ChartQA reconstruction | rFID: 7.70 -> 5.27 | Helped |
| reduce latent channel dimension from c=16 to c=8 (ablation) | CAT · ImageNet generation | FID: 4.56 -> 4.38 | Helped |
| reduce latent channel dimension from c=16 to c=4 (ablation) | CAT · ImageNet generation | FID: 4.56 -> 5.12 | Hurt |
| increase CAT generation token count from 256 to 1024 (ablation) | DiT-XL/2 with CAT · ImageNet generation | FID-50K: 5.02 -> 4.12 | Helped |
| reduce CAT generation token count from 256 to 64 (ablation) | DiT-XL/2 with CAT · ImageNet generation | FID-50K: 5.02 -> 5.83 | Hurt |
Surprising findings
- Larger latent channel c improves reconstruction but not generation; c=8 gave the best FID for both fixed and adaptive settings.
- JPEG file size as a complexity metric assigns nearly all CelebA and ChartQA images the highest 32x compression, which hurts reconstruction on text and face images.
Limitations
- Training data is proprietary (380M Shutterstock images), limiting reproducibility.
- Complexity thresholds and the 8/16/32 ratio set are hand-chosen; the LLM scorer's agreement with max acceptable ratio is only 62.39%.
- Evaluated only on continuous tokenizers and class-conditional ImageNet generation at 512px; discrete tokenizers and text-to-image generation are left as future work.
Experimental setup
Datasets: Shutterstock licensed images (380M) ImageNet COCO 2014
Benchmarks: COCO 2014 ImageNet CelebA ChartQA ImageNet 512 class-conditional generation
Metrics: rFID LPIPS PSNR FID sFID Inception Score precision recall throughput
Baselines: Fixed 8x VAE Fixed 16x VAE Fixed 32x VAE JPEG-adaptive VAE LDM VAE 8x/16x/32x DiT-XL with LDM VAE
Builds on: VAE U-Net Matryoshka representation learning DiT LDM VAE
Closest papers by meaning
- Single-pass Adaptive Image Tokenization for Minimum Program Search (NeurIPS 2025 · similarity 0.87)
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.86)
- Adaptive Length Image Tokenization via Recurrent Allocation (ICLR 2025 · similarity 0.86)
- Highly Compressed Tokenizer Can Generate Without Training (ICML 2025 · similarity 0.86)
- Language-Guided Image Tokenization for Generation (CVPR 2025 · similarity 0.85)
- FlexTok: Resampling Images into 1D Token Sequences of Flexible Length (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.