Research map / Image generation / Autoregressive and latent generation
ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ScaleDiff, a training-free, model-agnostic framework that extends pretrained SDXL and FLUX to 2048² and 4096² using Neighborhood Patch Attention (non-overlapping query patches with overlapping K/V neighborhoods), Latent Frequency Mixing, and Structure Guidance within an SDEdit-style upscaling pipeline.
Problem
Pretrained text-to-image diffusion models (U-Net and DiT) degrade when generating images beyond their training resolution, and existing training-free high-resolution methods are either computationally expensive (overlapping patch methods) or architecture-specific or incompatible with DiTs.
Approach
Replace overlapping MultiDiffusion patches with non-overlapping query patches in self-attention, each attending to an overlapping key/value neighborhood, so the self-attention cost and non-self-attention costs are not inflated; combine with latent frequency mixing of RGB-upsampled and latent-upsampled references and low-frequency structure guidance in latent space.
Key results
- On SDXL at 4096², ScaleDiff reaches FID 61.87 and runs in 113 s, the fastest among training-free methods and 8.9x faster than DemoFusion (1005 s).
- On FLUX at 4096², ScaleDiff gives a 3.1x speedup over direct inference (407 s vs 1251 s) with FID 64.06 vs 459.07 for direct.
- Ablation shows NPA reaches patch-level scores close to MultiDiffusion (FID 61.87 vs 61.71) while cutting FLUX time from 1148 s to 407 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ScaleDiff instead of DiffuseHigh | SDXL · SDXL 4096² | FID: 63.91 -> 61.87 | Helped |
| Use NPA instead of direct high-resolution attention (ablation) | ScaleDiff pipeline · SDXL 4096² ablation | Time (s): 185 -> 113 | Helped |
| Replace NPA with MultiDiffusion (ablation) | ScaleDiff pipeline · SDXL 4096² ablation | FID: 61.87 -> 61.71 | Helped |
| Use NPA instead of MultiDiffusion (ablation) | ScaleDiff pipeline · SDXL 4096² ablation | Time (s): 239 -> 113 | Helped |
| Add LFM and SG (ablation) | NPA · SDXL 4096² ablation | FID: 64.17 -> 61.87 | Helped |
| Add LFM (ablation) | NPA without LFM or SG · SDXL 4096² ablation | FIDp: 41.55 -> 39.49 | Helped |
| Add LFM (ablation) | NPA + SG · SDXL 4096² ablation | FID: 64.12 -> 61.87 | Helped |
| Add SG (ablation) | NPA without LFM or SG · SDXL 4096² ablation | FID: 64.17 -> 64.12 | Helped |
| Add SG (ablation) | NPA + LFM · SDXL 4096² ablation | FID: 62.34 -> 61.87 | Helped |
| Set τ to 400 rather than 700 (ablation) | ScaleDiff on SDXL · SDXL 4096² | FID: 63.61 -> 63.12 | Helped |
| Vary τ from 700 to 400 (ablation) | ScaleDiff on FLUX · FLUX 4096² | FID: 64.44 -> 63.59 | Helped |
| Vary τ across tested values (ablation) | ScaleDiff on SDXL · SDXL 4096² | FIDp: 37.65; 37.94; 38.77; 38.33; 39.38 | Mixed |
Limitations
- Performance is bounded by the capabilities of the underlying diffusion model.
- As a patch-based approach it can produce inconsistent local content in sharp close-up images and repetitive artifacts in background regions.
Experimental setup
Datasets: LAION-5B
Benchmarks: LAION-5B 1,000 sampled image-text pairs
Metrics: FID KID IS patch-level FID patch-level KID patch-level IS CLIP Score inference time
Baselines: SDXL Direct FLUX Direct BSRGAN OSEDiff UltraPixel ScaleCrafter HiDiffusion DiffuseHigh FreeScale AccDiffusion v2 DemoFusion
Builds on: MultiDiffusion SDEdit Structure Guidance DemoFusion
Closest papers by meaning
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.91)
- SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers (NeurIPS 2026 · similarity 0.90)
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.90)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.90)
- Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models (NeurIPS 2026 · similarity 0.90)
- SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.