Research map / Image generation / Fast diffusion sampling

Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget

CVPR 2025 · Image generation › Fast diffusion sampling · approach: Diffusion models as perception tools · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a 1.16B sparse diffusion transformer (MicroDiT) from scratch for $1,890 using 75% patch masking with a deferred patch-mixer, MoE layers, and synthetic images.

Problem

Training large-scale text-to-image diffusion transformers is extremely expensive (tens of thousands of GPU hours), which concentrates development among well-resourced actors; naive patch masking degrades quality at high masking ratios.

Approach

Apply a lightweight patch-mixer transformer to all patch embeddings before random masking (deferred masking), so unmasked patches retain whole-image context, enabling 75% masking with little degradation; combined with MoE layers, layer-wise scaling, and synthetic data for micro-budget training.

Key results

• MicroDiT (1.16B, 37M images) reaches 12.66 zero-shot FID-30K on COCO for $1,890 (2.6 8xH100 days), vs. 37.6 GPU days for PixArt-α.
• Deferred masking at 75% masking ratio yields FID 5.03 vs. 16.5 for naive masking (small-scale DiT-Tiny ablation on cifar-captions; no-masking FID 3.79).
• Training on 490M synthetic images at the same budget improves GenEval from 0.46 to 0.52 and is preferred by humans in at least 68% of comparisons.

Measured effects reported in the paper

Surprising findings

• Training only on real images yields nearly identical FID and GenEval to the real+synthetic mix, yet the mixed model is preferred about three times more by automated multimodal evaluation and over 65% by humans.
• Higher latent dimensionality hurts performance under micro-budget training.

Limitations

• Relies on the 75% masking ratio being the practical limit; performance drops at extremely high masking ratios (e.g., 87.5%).
• Synthetic data includes many early low-quality latent diffusion images (~10M), and results depend on public datasets whose quality varies.
• Cost estimates are based on a cloud GPU price per hour rather than actual expenditure.

Experimental setup

Datasets: CC12M SA1B TextCaps JourneyDB DiffusionDB Diffusion1B COYO-700M CIFAR-10

Benchmarks: COCO 2014 validation zero-shot FID-30K GenEval CLIP score

Metrics: FID CLIP-FID CLIP score GenEval accuracy human preference win rate

Baselines: Stable Diffusion 1.5 Stable Diffusion 2.1 PixArt-α Würstchen MaskDiT Dall-E.2 GigaGAN ImageN

Builds on: DiT MaskDiT expert-choice routing layer-wise scaling classifier-free guidance

Closest papers by meaning

• Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.90)
• SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer (ICML 2025 · similarity 0.89)
• PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (ICLR 2024 · similarity 0.89)
• On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.89)
• Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.89)
• SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.89)