Research map / Image generation / Autoregressive and latent generation
Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes self-guidance for masked generative models (MGMs) using a TOAST-based information bottleneck trained with an error-token-correction auxiliary task to semantically smooth VQ tokens, enabling high-temperature sampling with better quality-diversity on ImageNet.
Problem
Masked generative models underperform continuous diffusion models in FID and quality-diversity trade-off on class-conditional ImageNet image synthesis, and guidance methods from continuous diffusion do not directly transfer to discrete VQ token spaces.
Approach
Generalizes guidance formulations to discrete diffusion, and defines a semantic-smoothing information bottleneck for VQ tokens by fine-tuning a PEFT module (TOAST) with an error-token-correction auxiliary objective, then uses its smoothed predictions as guidance during MaskGIT sampling.
Key results
- On ImageNet 256x256 with 18 steps, the method reaches FID 3.22 and IS 263.9, versus MaskGIT's 6.56 and 203.6.
- On ImageNet 512x512, FID 5.38 (T=12) and 5.57 (T=18) versus MaskGIT's 8.48, with precision 0.88.
- With 5 sampling steps the method achieves FID 4.84 and IS 249.9, outperforming MaskGIT at 18 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add self-guidance | MaskGIT · ImageNet 256×256 | FID: 6.56 -> 3.22 | Helped |
| Add self-guidance | MaskGIT · ImageNet 256×256 | IS: 203.6 -> 263.9 | Helped |
| Add self-guidance | MaskGIT · ImageNet 512×512 | FID: 8.48 -> 5.57 | Helped |
| Add self-guidance | MaskGIT · ImageNet 512×512 | IS: 167.1 -> 233.2 | Helped |
| Use Gaussian blur in logit space for guidance (ablation) | MaskGIT · ImageNet 256×256 | FID: 8.32 -> 3.22 | Hurt |
| Use Gaussian blur in logit space for guidance (ablation) | MaskGIT · ImageNet 256×256 | IS: 231.2 -> 263.9 | Helped |
| Use self-attention value for adversarial masking (ablation) | MaskGIT · ImageNet 256×256 | FID: 4.73 -> 3.22 | Helped |
| Use self-attention value for adversarial masking (ablation) | MaskGIT · ImageNet 256×256 | IS: 177.3 -> 263.9 | Helped |
| Fine-tune TOAST with L_aux instead of L_mask (ablation) | MaskGIT with TOAST · ImageNet 256×256 | FID: 4.11 -> 3.22 | Helped |
| Fine-tune TOAST with L_aux instead of L_mask (ablation) | MaskGIT with TOAST · ImageNet 256×256 | IS: 238.4 -> 263.9 | Helped |
| Use guidance scale 1.0 rather than strong guidance (s>3) (ablation) | Self-guided MaskGIT · ImageNet 256×256 | FID | Helped |
| Increase sampling temperature to τ=25 with self-guidance (ablation) | MaskGIT · ImageNet 256×256 | FID | Helped |
Surprising findings
- Guidance enables much higher sampling temperatures (tau=25 at 256px, 45 at 512px) than MaskGIT's tau=4.5 while preserving quality.
- Plain Gaussian blur on VQ tokens yields marginal improvement, motivating the learned semantic-smoothing auxiliary task.
Limitations
- Only evaluated on class-conditional ImageNet; text-conditional and large-scale models left for future work.
- Strong guidance (s>3) or very high temperature causes oversaturated colors and collapsed structure.
- Guidance roughly doubles model inferences per step.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 ImageNet 512x512
Metrics: FID IS Precision Recall NFE
Baselines: MaskGIT Token-Critic DPC LDM-4 with CFG ADM DiT-L/2 VQ-Diffusion Improved VQ-Diffusion BigGAN-deep GigaGAN VQGAN
Builds on: MaskGIT TOAST Self-guidance (SAG) Classifier-free guidance Token-Critic
Closest papers by meaning
- Fixed-Point Masked Generative Modeling (NeurIPS 2026 · similarity 0.88)
- Simple and Fast Distillation of Diffusion Models (NeurIPS 2024 · similarity 0.87)
- Devil is in the Details: Density Guidance for Detail-Aware Generation with Flow Models (ICML 2025 · similarity 0.87)
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.87)
- Consistency Posterior Sampling for Diverse Image Synthesis (CVPR 2025 · similarity 0.87)
- CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.