Research map / Image generation / Editing and style transfer
Highly Compressed Tokenizer Can Generate Without Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a pretrained 1D tokenizer (TiTok VQ-LL-32) supports simple token copy-paste editing and gradient-based test-time token optimization with CLIP or reconstruction losses, enabling training-free text-guided editing, inpainting, and class-conditional generation.
Problem
Whether highly compressed 1D image tokenizers can be used as generative models directly, enabling image editing, inpainting, and generation without training any dedicated generative model.
Approach
Treat a pretrained 1D VQ tokenizer's discrete latent space as an expressive generative space: perform copy-and-paste of tokens for appearance edits and gradient-based test-time optimization of continuous pre-quantization features (with straight-through VQ) under CLIP or L1 reconstruction objectives, plus tweaks such as crop-averaged loss, token noise, EMA and L2 regularization.
Key results
- Test-time token optimization with VQ-LL-32 and 1000 random seed images reaches FID 8.2 (50k) and IS 182, comparable to or better than several trained unconditional/semi-parametric generators (e.g., RCDM 19.0, IC-GAN 15.6, RDM-IN 5.9 with trained models).
- Generation quality degrades with more tokens or larger codebooks: VQ-LL-32 FID-5k 15.1 vs VQ-BB-64 16.3 and VQ-BL-128 22.3.
- Replacing VQ with continuous VAE tokens raises FID-5k from 15.1 to 33.2; the 2D MaskGIT-VQGAN gives FID-5k 34.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the 32-token VQ tokenizer rather than the 64- or 128-token variants (ablation) | TiTok test-time generation · ImageNet generation | FID-5k: 16.3 -> 15.1; 18.6 -> 15.1; 22.3 -> 15.1 | Helped |
| Use vector quantization instead of optimizing continuous features without VQ (ablation) | TiTok VQ-LL-32 · ImageNet generation | FID-5k: 17.1 -> 15.1 | Helped |
| Use discrete VQ tokens instead of continuous VAE tokens (ablation) | TiTok LL-32 test-time generation · ImageNet generation | FID-5k: 33.2 -> 15.1 | Helped |
| Use compressed 1D tokens instead of MaskGIT's 2D discrete tokens (ablation) | Test-time token optimization · ImageNet generation | FID-5k: 34.3 -> 15.1 | Helped |
| Optimize seed tokens rather than use reconstructed seed images directly (ablation) | TiTok VQ-LL-32 · ImageNet generation | FID-5k: 73.7 -> 15.1 | Helped |
| Add random crops to the CLIP loss (ablation) | TiTok VQ-LL-32 token optimization · ImageNet generation | Inception Score: 140 -> 157 | Helped |
| Add random crops to the CLIP loss (ablation) | TiTok VQ-LL-32 token optimization · ImageNet generation | FID-5k: 14.8 -> 15.3 | Hurt |
| Return the exponential moving average of token iterates (ablation) | TiTok VQ-LL-32 token optimization · ImageNet generation | FID-5k: 15.3 -> 15.1 | Helped |
| Reduce optimization by 100 iterations (ablation) | TiTok VQ-LL-32 token optimization · ImageNet generation | FID-5k: 15.1 -> 15.2 | Hurt |
| Add noise to tokens at the beginning of each optimization iteration (ablation) | TiTok VQ-LL-32 token optimization · ImageNet generation | FID-5k: 15.1 -> 14.5 | Helped |
| Add weak L2 token regularization (ablation) | TiTok VQ-LL-32 token optimization · ImageNet generation | Inception Score: 160 -> 182 | Helped |
| Add weak L2 token regularization (ablation) | TiTok VQ-LL-32 token optimization · ImageNet generation | FID-5k: 15.1 -> 14.6 | Hurt |
Surprising findings
- Adversarially maximizing CLIP score on pixels yields high CLIP similarity (0.59) but near-zero IS and poor SigLIP alignment, showing CLIP guidance can be fooled.
- Even crude single-token copy-and-paste from a reference image can transfer global appearance attributes such as background blur and lighting.
Limitations
- Pose and semantic attribute transfer by token copying works only to a limited extent.
- Tokenizer is trained only on ImageNet, so out-of-domain classes and styles give inaccurate or implausible results.
- Number of optimization iterations must be chosen per example; no adaptive stopping criterion is provided.
Experimental setup
Datasets: ImageNet ILSVRC2012 training subset (seed images) ImageNet validation set (token analysis)
Benchmarks: ImageNet 50k class-conditional generation (FID, IS)
Metrics: FID-5k FID-50k IS CLIP similarity SigLIP similarity
Baselines: ADDP RCG-G RCDM IC-GAN RDM-IN VAE-LL-32 MaskGIT-VQGAN adversarial CLIP pixel optimization
Builds on: TiTok VQGAN-CLIP CLIP
Closest papers by meaning
- An Image is Worth 32 Tokens for Reconstruction and Generation (NeurIPS 2024 · similarity 0.89)
- FlexTok: Resampling Images into 1D Token Sequences of Flexible Length (ICML 2025 · similarity 0.89)
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.89)
- Language-Guided Image Tokenization for Generation (CVPR 2025 · similarity 0.88)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.88)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.