Research map / Image generation / Autoregressive and latent generation
Boosting Generative Image Modeling via Joint Image-Feature Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReDi, which jointly diffuses VAE latents and PCA-reduced DINOv2 features in a single DiT/SiT, plus Representation Guidance at inference, speeding ImageNet 256 generation convergence.
Problem
Latent diffusion models learn low-level VAE latents without explicit semantic structure, and prior work (REPA) only aligns diffusion features to external self-supervised features via distillation; the goal is to improve generation quality and convergence speed by integrating representation learning
Approach
Jointly diffuse VAE image latents and PCA-reduced DINOv2 features within the same diffusion/flow process, fusing both into one token sequence with minimal DiT/SiT changes, and at inference guide sampling with the model's own semantic-feature noise predictions (Representation Guidance), avoiding auxiliary distillation losses.
Key results
- ReDi DiT-XL/2 reaches FID 8.7 at 400K iterations without CFG, versus 9.6 for baseline DiT-XL/2 at 7M iterations.
- SiT-XL/2 with ReDi and CFG reaches FID 1.72 at 350 epochs, versus 2.06 for SiT-XL/2 at 1400 epochs.
- Representation Guidance lowers DiT-XL/2 w/ ReDi FID from 8.7 to 5.9 without CFG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly model VAE latents and DINOv2 features instead of using the DiT-L/2 baseline | DiT-L/2 · ImageNet 256×256, no CFG, 400K iterations | FID: 23.2 -> 10.5 | Helped |
| use ReDi rather than REPA | DiT-L/2 · ImageNet 256×256, no CFG, 400K iterations | FID: 15.6 -> 10.5 | Helped |
| jointly model VAE latents and DINOv2 features instead of the SiT-XL/2 baseline | SiT-XL/2 · ImageNet 256×256, no CFG, 400K iterations | FID: 17.2 -> 7.5 | Helped |
| use ReDi rather than REPA | SiT-XL/2 · ImageNet 256×256, no CFG, 400K iterations | FID: 7.9 -> 7.5 | Helped |
| jointly model VAE latents and DINOv2 features instead of the unconditional DiT baseline | DiT-B/2 · ImageNet 256×256, unconditional generation, 400K steps | FID: 69.3 -> 51.7 | Helped |
| add Representation Guidance (ablation) | DiT-XL/2 + ReDi · ImageNet 256×256, conditional generation, 400K steps | FID: 8.7 -> 5.9 | Helped |
| add Representation Guidance (ablation) | DiT-XL/2 + ReDi · ImageNet 256×256, unconditional generation, 400K steps | FID: 25.1 -> 22.6 | Helped |
| add ReDi to REPA (ablation) | SiT-XL/2 + REPA · ImageNet 256×256, no CFG | FID: 5.9 -> 3.5 | Helped |
| use separate tokens instead of merged tokens (ablation) | DiT-B/2 + ReDi · ImageNet 256×256, no CFG | FID: 25.7 -> 24.7 | Helped |
| use separate tokens instead of merged tokens (ablation) | DiT-B/2 + ReDi · ImageNet 256×256, no CFG | throughput: 4.51 -> 2.26 | Hurt |
| apply CFG to VAE latents only rather than both VAE and DINOv2 representations (ablation) | SiT-XL/2 + ReDi · ImageNet 256×256, CFG | FID: 2.86 -> 2.39 | Helped |
| increase the number of DINOv2 principal components up to r=8 (ablation) | DiT-B/2 + ReDi · ImageNet 256×256, no CFG | FID | Mixed |
Surprising findings
- Applying CFG to DINOv2 features as well as VAE latents hurts performance (FID 2.86 vs 2.39 for VAE-only CFG).
Limitations
- Only a single visual representation (DINOv2) is explored.
- PCA is used for dimensionality reduction; learned compression such as an autoencoder is left for future work.
- Separate-token fusion doubles token count and compute.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256
Metrics: FID sFID IS Precision Recall
Baselines: DiT-XL/2 SiT-XL/2 REPA MaskDiT MDT VAR MAR FasterDiT
Builds on: DiT SiT REPA classifier-free guidance
Closest papers by meaning
- Nested Diffusion Models Using Hierarchical Latent Priors (CVPR 2025 · similarity 0.92)
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.92)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.90)
- End-to-End Training for Unified Tokenization and Latent Denoising (NeurIPS 2026 · similarity 0.90)
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.90)
- Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.