Research map / Image generation / Autoregressive and latent generation
DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCTdiff, an end-to-end diffusion model that generates images directly in JPEG-style block DCT coefficient space (YCbCr, chroma subsampling, zigzag truncation), outperforming pixel diffusion and SD-VAE latent diffusion at lower training cost, and provides a theory of image diffusion as spectral autoregression.
Problem
Pixel-space diffusion is expensive and hard to scale to high resolution, while latent diffusion relies on a costly pretrained VAE; the paper asks whether diffusion can model images efficiently directly in a compressed frequency (DCT) space.
Approach
Model images as JPEG-style block DCT coefficients (YCbCr, 2x chroma subsampling, zigzag truncation of high-frequency coefficients) with standard UViT/DiT backbones and diffusion objectives, adding entropy-consistent scaling, SNR scaling, and entropy-based frequency reweighting; also proves that diffusion acts as spectral autoregression and derives a DCT upsampling rule.
Key results
- DCTdiff beats UViT pixel-space on most datasets, e.g. CIFAR-10 FID 5.02 vs 6.23 at NFE=100 with DDIM, and FFHQ 128 FID 8.22 vs 13.87 with DDIM at NFE=100.
- On AFHQ 512, DCTdiff reaches FID 8.76 vs 10.86 for latent UViT (NFE=100, DPM-Solver), at about one quarter of the training GFLOPs (133 vs 575).
- DCT compression with rFID<0.5 gives 4x compression at 256x256 and 7.11x at 512x512 on FFHQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| model images in DCT space instead of pixel space | UViT · CIFAR-10, DDIM, NFE 100 | FID-50k: 6.23 -> 5.02 | Helped |
| model images in DCT space instead of pixel space | UViT · CelebA 64, DPM-Solver, NFE 100 | FID-50k: 1.57 -> 1.71 | Hurt |
| model images in DCT space instead of pixel space | DiT · CelebA 64, DDPM, NFE 100 | FID-50k: 5.11 -> 3.84 | Helped |
| use DCT diffusion instead of latent diffusion | UViT · FFHQ 256, DPM-Solver, NFE 100 | FID-50k: 4.26 -> 5.08 | Hurt |
| use DCT diffusion instead of latent diffusion | UViT · FFHQ 512, DPM-Solver, NFE 100 | FID-50k: 10.89 -> 7.07 | Helped |
| use DCT diffusion instead of latent diffusion | UViT · AFHQ 512, DPM-Solver, NFE 100 | FID-50k: 10.86 -> 8.76 | Helped |
| train in DCT space instead of pixel space | UViT · CelebA 64 | training steps: 400k -> 250k | Helped |
| train in DCT space instead of latent space | UViT · AFHQ 512 | GFLOPs: 575 -> 133 | Helped |
| use DCT upsampling instead of pixel upsampling | image upsampling · image upsampling evaluation | FID: 12.53 -> 9.79 | Helped |
| use DCT diffusion instead of pixel diffusion | UViT · CelebA 64, DPM-Solver, NFE 100 | FID-50k | Hurt |
Surprising findings
- Pixel-space UViT on FFHQ 256 reached FID 120, while DCTdiff scales to 512x512 without any VAE.
- DCTdiff is slower than latent UViT at high NFE on AFHQ 512 (e.g. 47.8 vs 20.2 min at NFE=100), despite lower training cost.
Limitations
- Frequency-oriented Transformer architecture not explored; only UViT and DiT backbones were tested.
- Image generation at 1024x1024 resolution not explored.
- Inference is slower than latent UViT at high NFE because DCTdiff uses more tokens.
Experimental setup
Datasets: CIFAR-10 CelebA 64 ImageNet 64 FFHQ 128 FFHQ 256 FFHQ 512 AFHQ 512
Benchmarks: CIFAR-10 CelebA 64 ImageNet 64 FFHQ 128 FFHQ 256 FFHQ 512 AFHQ 512
Metrics: FID-50k CMMD IS precision recall rFID wall-clock inference time GFLOPs training steps
Baselines: UViT (pixel) UViT (latent, SD-VAE) DiT (pixel) DPM-Solver DDIM sampler
Builds on: UViT DiT DPM-Solver VP-SDE score-based diffusion JPEG
Closest papers by meaning
- DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents (ICML 2024 · similarity 0.89)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.89)
- Diffusion Models Without Attention (CVPR 2024 · similarity 0.89)
- Relay Diffusion: Unifying diffusion process across resolutions for image synthesis (ICLR 2024 · similarity 0.89)
- Denoising Diffusion Bridge Models (ICLR 2024 · similarity 0.89)
- Dynamic Diffusion Transformer (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.