Research map / Image generation / Autoregressive and latent generation
Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models
Bowei Chen, Sai Bi, Hao Tan, He Zhang, Tianyuan Zhang, Zhengqi Li, Yuanjun Xiong, Jianming Zhang, Kai Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align pretrained DINOv2 encoder instead of semantic regularization (VA-VAE) | Tokenizer · ImageNet 256 f16d32, w/ CFG | gFID: 3.13 -> 2.17 | Improved |
| Align pretrained encoder instead of semantic regularization | Tokenizer · ImageNet 256 unconditional | gFID: 19.12 -> 11.80 | Improved |
| Add semantic preservation loss in stage 2 (ablation) | AlignTok · ImageNet 256 | gFID: 3.05 -> 2.19 | Improved |
| Raise semantic preservation weight to 5 or 10 (ablation) | AlignTok · ImageNet 256 | rFID / gFID: rFID 0.36 -> 0.59; gFID 2.19 -> 2.59 | Mixed |
| Apply semantic loss before the adapter instead of on latent (ablation) | AlignTok · ImageNet 256 | gFID: 2.19 -> 2.83 | Hurt |
| Replace L2 semantic loss with cosine loss (ablation) | AlignTok · ImageNet 256 | gFID: 2.19 -> 2.23 | No clear effect |
| Use LoRA instead of full fine-tuning in stage 2 (ablation) | AlignTok · ImageNet 256 | rFID / gFID: rFID 0.36 -> 1.35; gFID 2.19 -> 2.97 | Hurt |
| Add EMA in stage 2 (ablation) | AlignTok · ImageNet 256 | gFID: 2.24 -> 2.19 | Improved |
| Use a larger decoder (ablation) | AlignTok · ImageNet 256 | gFID: 2.19 -> 2.52 | Hurt |
| Add stage 2 perceptual alignment to stage 1 (ablation) | AlignTok · ImageNet 256 | rFID: 1.63 -> 0.36 | Improved |
| Add stage 3 decoder refinement (ablation) | AlignTok · ImageNet 256 | rFID / gFID: rFID 0.36 -> 0.26; gFID 2.19 -> 2.17 | Improved |
| Use DINOv2 instead of MAE or SigLIP 2 (ablation) | AlignTok · ImageNet 256 | gFID: MAE 3.12, SigLIP 2 2.22 -> 2.19 | Improved |
Closest papers by meaning
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.92)
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.91)
- End-to-End Training for Unified Tokenization and Latent Denoising (NeurIPS 2026 · similarity 0.90)
- Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.90)
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.90)
- DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
