Research map / Image generation / Autoregressive and latent generation
ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ScaleCrafter, a tuning-free method that adapts pre-trained Stable Diffusion U-Nets to generate ultra-high-resolution images by dynamically re-dilating convolutions at inference, plus convolution dispersion and noise-damped classifier-free guidance.
Problem
Pre-trained diffusion models produce object repetition and malformed structures when sampled at resolutions well above their training size, and prior attention-scaling and joint-diffusion methods do not fix this.
Approach
Attributes object repetition to the limited receptive field of U-Net convolution kernels rather than attention token count, and dynamically dilates (re-dilates) convolutions only at inference, with timestep- and layer-aware schedules, fractional dilation, convolution dispersion via least-squares kernel enlargement, and noise-damped CFG that uses a less-dilated model for the denoising base.
Key results
- At 16x pixels on SD 1.5, FID_r drops from 111.34 (direct inference) to 78.22 with ScaleCrafter.
- On T2V (LVDM) at 4x, FVD_r falls from 674.14 to 418.80 versus direct inference.
- Against SD 2.1 upscaler at 16x, users preferred ScaleCrafter's texture definition 62% vs 38%, though SD-SR has better FID/KID at 4x (12.59 vs 20.88).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ScaleCrafter instead of Attn-SF | SD 1.5 · 16× 1:1 | FID_r: 110.10 -> 78.22 | Helped |
| use ScaleCrafter instead of Attn-SF | SD 2.1 · 16× 1:1 | FID_r: 104.34 -> 59.40 | Helped |
| use ScaleCrafter instead of Attn-SF | SD XL 1.0 · 16× 1:1 | FID_r: 153.68 -> 131.03 | Helped |
| replace SD+SR with ScaleCrafter | SD 2.1 · 4× | FID_r: 12.59 -> 20.88 | Hurt |
| replace SD+SR with ScaleCrafter | SD 2.1 · 16× | Texture Definition: 38% -> 62% | Helped |
| use ScaleCrafter instead of direct inference | LVDM · Webvid-10M, 4× | FVD_r: 674.14 -> 418.80 | Helped |
| add timestep-wise progressive re-dilation (ablation) | ScaleCrafter · SD 1.5, 16× 1:1 | FID_r: 94.90 -> 78.22 | Helped |
| add convolution dispersion (ablation) | ScaleCrafter · SD 1.5, 16× 1:1 | FID_r: 106.18 -> 78.22 | Helped |
| add noise-damped classifier-free guidance (ablation) | ScaleCrafter · SD 1.5, 16× 1:1 | FID_r: 112.15 -> 78.22 | Helped |
Surprising findings
- Re-dilating only the attention token grouping while keeping convolution unchanged does not fix repetition, pointing to convolution kernels as the primary cause.
- Attention-scaling (Attn-SF) gives nearly no improvement over direct inference in the reported tables.
Limitations
- Slightly worse FID/KID than the SD upscaler at 4x resolution
- Hyperparameters (dilation scales, blocks, tau) are chosen per resolution and model
- Ultra-high resolution quality degrades without dispersion and noise-damped CFG
Experimental setup
Datasets: LAION-5B WebVid-10M
Benchmarks: LAION-5B subset (FID/KID) WebVid-10M (FVD/KVD)
Metrics: FID KID FVD KVD user preference texture definition
Baselines: Direct inference Attn-SF SD 2.1 upscaler (SD-SR) MultiDiffusion SyncDiffusion
Builds on: Attn-SF MultiDiffusion SyncDiffusion classifier-free guidance dilated convolution
Closest papers by meaning
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.92)
- ResMaster: Mastering High-Resolution Image Generation via Structural and Fine-Grained Guidance (AAAI 2025 · similarity 0.90)
- UltraPixel: Advancing Ultra High-Resolution Image Synthesis to New Peaks (NeurIPS 2024 · similarity 0.90)
- Native-Resolution Image Synthesis (NeurIPS 2025 · similarity 0.89)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.89)
- Hierarchical Patch Diffusion Models for High-Resolution Video Generation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.