Research map / Architectures and efficiency / Attention and transformers
DiC: Rethinking Conv3x3 Designs in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds DiC, a purely 3x3-convolutional U-Net diffusion model with sparse skips and improved conditioning that beats DiT and U-ViT on ImageNet with higher throughput.
Problem
Diffusion transformers rely on costly self-attention, causing slow inference; the paper asks whether a purely convolutional (3x3) diffusion model can match transformer-based diffusion models in quality while being faster.
Approach
Use stride-1 3x3 convolution blocks in an encoder-decoder hourglass (U-Net) with sparse strided skip connections, stage-specific condition embeddings, mid-block condition injection, AdaLN-style conditional gating, and GELU activations to build a scaled, fast pure-CNN diffusion backbone.
Key results
- DiC-XL reaches FID 13.11 vs DiT-XL/2 20.05 at 400K iterations on ImageNet 256x256 without CFG, with throughput 313.7 vs 66.8.
- DiC-H achieves FID 11.36 at 400K and 8.96 at 800K without CFG; with CFG, DiC-H reaches FID 2.25 at 2M iterations vs DiT-XL/2 2.27 at 7M.
- DiC-XL with cfg=1.5 achieves FID 3.89 at 400K iterations vs DiT-XL/2 6.24.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DiT-S/2 with DiC-S | DiC-S · ImageNet 256×256, 400K | FID: 67.40 -> 58.68 | Helped |
| replace DiT-B/2 with DiC-B | DiC-B · ImageNet 256×256, 400K | FID: 42.84 -> 32.33 | Helped |
| replace DiT-XL/2 with DiC-XL | DiC-XL · ImageNet 256×256, 400K | FID: 20.05 -> 13.11 | Helped |
| use DiC-XL instead of DiT-XL/2 | DiC-XL · ImageNet 256×256, 400K | throughput: 66.8 -> 313.7 | Helped |
| use DiC-XL instead of DiT-XL/2 | DiC-XL · ImageNet 512×512, 400K | FID: 20.94 -> 15.32 | Helped |
| use DiC-XL instead of DiT-XL/2 | DiC-XL · ImageNet 256×256, CFG, 400K | FID: 6.24 -> 3.89 | Helped |
| replace isotropic Conv3x3 architecture with U-Net hourglass (ablation) | Conv3x3 diffusion model · ImageNet 256×256, 200K, CFG=1.5 | FID: 29.31 -> 14.65 | Helped |
| replace dense U-Net skips with sparse skips (ablation) | Conv3x3 U-Net · ImageNet 256×256, 200K, CFG=1.5 | FID: 14.65 -> 11.49 | Helped |
| add stage-specific condition embeddings (ablation) | Sparse-Skip U-Net · ImageNet 256×256, 200K, CFG=1.5 | FID: 11.49 -> 10.07 | Helped |
| inject conditions mid-block (ablation) | Sparse-Skip U-Net with stage-specific embeddings · ImageNet 256×256, 200K, CFG=1.5 | FID: 10.07 -> 8.80 | Helped |
| add conditional gating (ablation) | Sparse-Skip U-Net with mid-block injection · ImageNet 256×256, 200K, CFG=1.5 | FID: 8.80 -> 6.54 | Helped |
| replace activation functions with GELU (ablation) | DiC · ImageNet 256×256, 200K, CFG=1.5 | FID: 6.54 -> 6.26 | Helped |
Surprising findings
- A pure 3x3-convolution diffusion model with no self-attention outperforms isotropic DiT-style transformers, with the encoder-decoder hourglass being essential for the limited receptive field of 3x3 convolutions.
- Isotropic Conv3x3 models performed much worse (FID 29.31) than the U-Net hourglass (FID 14.65), reversing the usual scalability intuition from transformers.
Limitations
- Performance of isotropic pure-convolution variants is weak, so the hourglass design is required.
- Results compare against DiT-style baselines under aligned but limited training budgets; many comparisons are without classifier-free guidance.
- Throughput comparisons depend on hardware and implementation details (Winograd, flash attention in baselines).
Experimental setup
Datasets: ImageNet 2012
Benchmarks: ImageNet 256x256 ImageNet 512x512
Metrics: FID IS throughput FLOPs
Baselines: DiT-S/2 DiT-B/2 DiT-XL/2 U-ViT-XL U-ViT-H PixArt-α-XL/2 DiffiT-XL/2 DiT-LLaMA
Builds on: DiT U-ViT ADM Latent Diffusion RepVGG ConvNeXt
Closest papers by meaning
- DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling (NeurIPS 2025 · similarity 0.92)
- Why DiT Models Underperform as Representation Learners without Long Skip Connections (NeurIPS 2026 · similarity 0.89)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.88)
- Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture (CVPR 2024 · similarity 0.88)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.