Research map / Image generation / Autoregressive and latent generation
Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows pixel-space diffusion can match latent diffusion on ImageNet512 by combining a sigmoid loss weighting with a shifted bias, Residual U-ViT architecture with fewer skip connections, and flop-heavy scaling (plus distillation), reaching FID 1.48 (1.50 distilled).
Problem
End-to-end pixel-space diffusion has lagged behind latent diffusion at high resolution (e.g., ImageNet512), largely due to loss weighting and architecture choices that do not scale to high resolutions.
Approach
Use the sigmoid loss weighting (monotonically increasing in x-space MSE) with a resolution-dependent negative bias, a Residual U-ViT that keeps only one skip connection per downsampling level, and flop-heavy scaling (smaller patches rather than more channels) to make single-stage pixel diffusion competitive with latent models.
Key results
- SiD2 Flop Heavy reaches FID 1.48 on ImageNet512 (1.50 with 16-step distillation), improving pixel-space SOTA from 2.65 (VDM++).
- SiD2 Flop Heavy achieves FID 1.26 on ImageNet128 and SiD2 Flop Heavy 1.38 on ImageNet256, state-of-the-art among listed methods.
- On Kinetics600 video prediction, SiD2 reaches FVD50k 2.3, better than W.A.L.T (3.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sigmoid loss weighting instead of EDM-monotonic or other tested weightings (ablation) | Pixel-space diffusion on ImageNet512 · ImageNet512 | FID | Helped |
| Shift sigmoid weighting bias to -3 rather than -2 or -1 (ablation) | SiD2 · ImageNet512 | FID: 1.55 -> 1.48; 1.75 -> 1.48 | Helped |
| Use sigmoid loss weighting with bias 0 rather than eps-mse (ablation) | SiD2 video generation · Kinetics600 | FVD50k: 5.7 -> 2.3 | Helped |
| Set sigmoid bias to +2 instead of 0 (ablation) | SiD2 video generation · Kinetics600 | FVD50k: 2.3 -> 3.2 | Hurt |
| Set sigmoid bias to -2 instead of 0 (ablation) | SiD2 video generation · Kinetics600 | FVD50k: 2.3 -> 2.5 | Hurt |
| Scale with more tokens using 2 x 2 patching rather than doubling channels (ablation) | SiD2 on ImageNet512 · ImageNet512 | FID: 1.9 -> 1.54 | Helped |
| Add distribution augmentation to the model-size-2x variant (ablation) | SiD2 model size 2x · ImageNet512 | FID: 1.9 -> 1.6 | Helped |
| Remove blockwise skip-connections from the small model (ablation) | SiD2 small · ImageNet512 | FID: 0.2 FID | Hurt |
| Remove blockwise skip-connections from the small model (ablation) | SiD2 small · ImageNet256 | FID: 0.05 FID | Hurt |
| Remove blockwise skip-connections from larger models (ablation) | SiD2 larger models · ImageNet512 | FID | No clear effect |
| Move blocks from encoder to decoder (E1-D5 instead of E3-D3) (ablation) | Residual U-ViT · ImageNet | FID | Helped |
| Use an encoder-heavy architecture (E5-D1) rather than the E3-D3 baseline (ablation) | Residual U-ViT · ImageNet | FID | Hurt |
Surprising findings
- Removing blockwise skip connections does not hurt larger models and can slightly improve them.
- Flop-heavy scaling (smaller patch size) outperforms doubling channels, which suffers from overfitting on ImageNet.
Limitations
- Latent diffusion (EDM2) still shows slightly better scaling in quality versus compute.
- Pixel-space model is more expensive in FLOPs per forward pass than latent models at high resolution.
- Video and text-to-image results are less thoroughly compared; MSCOCO zero-shot FID is weakly correlated with perceived quality.
Experimental setup
Datasets: ImageNet Kinetics600 MSCOCO
Benchmarks: ImageNet128 ImageNet256 ImageNet512 Kinetics600 MSCOCO
Metrics: FID FID30k FVD50k
Baselines: ADM RIN SD, U-ViT-L VDM++ EDM2-XXL DiT-XL/2 PaGoDA StyleGAN-XL MAGVIT-v2 DMD2 W.A.L.T
Builds on: VDM++ simple diffusion U-ViT EDM2 moment matching distillation
Closest papers by meaning
- PixelDiT2: Representation-Grounded Pixel Diffusion Transformers (NeurIPS 2026 · similarity 0.93)
- Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models (CVPR 2024 · similarity 0.91)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.91)
- Diffusion Models Without Attention (CVPR 2024 · similarity 0.91)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.90)
- Unleashing the Power of One-Step Diffusion based Image Super-Resolution via a Large-Scale Diffusion Discriminator (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.