Research map / Image generation / Autoregressive and latent generation
LMD: Faster Image Reconstruction with Latent Masking Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LMD, a latent masking diffusion framework that runs a ViT-based masked autoencoder with progressively increasing mask ratios inside a VQ-GAN latent space to cut image reconstruction training time.
Problem
Diffusion models and masked autoencoders both incur very high training cost for image reconstruction: diffusion from long pixel-space step chains, MAEs from pixel-level modeling and a fixed high mask ratio (0.75).
Approach
Combine a pretrained VQ-GAN latent space projector with a ViT masked autoencoder whose mask ratio is increased over training by three schedulers (uniform, piecewise, cosine), so reconstruction proceeds from easy to hard without sequential denoising steps.
Key results
- On ImageNet-1K, LMD-CS reaches MLT 6.92 and MLI 2.65 versus MAE's 17.11 and 6.53, and the paper reports about 3x faster training to the same accuracy.
- On LSUN-Bedrooms 256x256, LMD achieves MIT 2.45, MLT 7.06 and MLI 2.88, beating LDM (2.86, 15.84, 5.54) on all three time metrics.
- On CC generative evaluation, LMD-CS obtains FID 6.2, CLIP-score 0.26 and LPIPS 0.27, better than SD-v1.4 (17.01) and Muse base (6.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cosine-based masking scheduling instead of piecewise scheduling (ablation) | LMD · ImageNet-1K | MLT: 8.37 -> 6.92 | Helped |
| Use cosine-based masking scheduling instead of piecewise scheduling (ablation) | LMD · ImageNet-1K | MAT@1: 0.158 -> 0.135 | Helped |
| Use cosine-based masking scheduling instead of piecewise scheduling (ablation) | LMD · ImageNet-1K | MAT@5: 0.137 -> 0.102 | Helped |
| Add a latent space projector (ablation) | LMD · ImageNet-1K | Overall mean time-consumption: 2.81 -> 2.48 | Helped |
| Use MAE encoder-decoder blocks instead of U-Net (ablation) | LMD · ImageNet-1K | Overall mean time-consumption: 5.18 -> 2.48 | Helped |
| Use masking diffusion scheduling instead of a fixed 0.75 mask ratio (ablation) | LMD · ImageNet-1K | Overall mean time-consumption: 3.75 -> 2.48 | Helped |
| Use latent masking diffusion rather than MAE | ImageNet-1K · ImageNet-1K | MAT@1: 0.374 -> 0.135 | Helped |
| Use latent masking diffusion rather than LDM | ImageNet generation · LSUN-Bedrooms 256 x 256 | MLT: 15.84 -> 7.06 | Helped |
| Use latent masking diffusion rather than Muse_base | generative evaluation · CC | FID: 6.8 -> 6.2 | Helped |
Surprising findings
- Removing the latent space projector only added 0.33 to the mean delay, suggesting the ViT partly offsets the latent-space speedup, though it may still reduce GPU memory use.
Limitations
- Gains partly rely on custom time metrics (MIT, MLT, MLI) rather than standard wall-clock comparisons, and the lowest iteration time is held by UM-MAE with a low mask ratio.
- The VQ-GAN latent projector can fail on images with unseen feature distributions, requiring an extra pixel reconstruction fine-tuning step.
Experimental setup
Datasets: ImageNet-1K LSUN-Bedrooms CC
Benchmarks: ImageNet-1K LSUN-Bedrooms CC
Metrics: mean iteration time mean loss-decrease time mean loss-decrease iterations MAT@1 MAT@5 FID CLIP-score LPIPS
Baselines: DDPM iDDPM DDIM LDM MAE SimMIM GreenMIM UM-MAE SD-v1.4 Muse
Builds on: MAE VQ-GAN Stable Diffusion DDPM BERT
Closest papers by meaning
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.90)
- OSCAR: One-Step Diffusion Codec Across Multiple Bit-rates (NeurIPS 2025 · similarity 0.89)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.88)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.88)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.87)
- Cross-view Masked Diffusion Transformers for Person Image Synthesis (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.