Research map / Image generation / Autoregressive and latent generation

FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion

CVPR 2025 · Image generation › Autoregressive and latent generation · approach: High-resolution diffusion synthesis · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Training-free, single-pass method that lets pretrained latent diffusion models (SDXL, HiDiffusion) generate higher-resolution images by combining Fourier-domain frequency modulation of the diffuse-denoise process with attention modulation from native-resolution attention maps.

Problem

Pretrained diffusion models produce repetitive patterns and structural distortions when sampled at resolutions beyond their training resolution, and prior fixes are either artifact-prone or incur large latency from patch-based generation.

Approach

Modulate the high-resolution denoising in the Fourier domain by taking low-frequency components from the diffused native-resolution latent (time-aware high-pass filter), and modulate up-block self-attention maps with upsampled native-resolution attention maps, all without training or architecture changes.

Key results

• At 4x4 scale (4096x4096) with SDXL, FAM diffusion reduces FIDc from 74.37 to 43.65 and raises CLIP score from 26.70 to 32.33 versus direct SDXL inference, at 6.1 vs 5.4 min latency.
• At 2x2 scale, SDXL + FAM diffusion achieves FIDc 33.96 vs DemoFusion 36.75, with latency 1 min vs DemoFusion 2.5 min.
• Adding FAM to HiDiffusion at 4x4 improves FIDc from 156.98 to 44.65 and CLIP from 24.32 to 32.31 with only 0.3 min extra latency.

Measured effects reported in the paper

Surprising findings

• Skip-residual steering from the native-resolution image (DemoFusion-style) yields inconsistent structures, which the frequency-based formulation avoids.
• Applying attention modulation at all UNet layers degrades results due to over-regularization; restricting it to up-block layers works better.

Limitations

• Metrics FID/KID downsample high-resolution images, so they miss fine-detail quality; the authors rely on patch-level FIDc/KIDc.
• Evaluation is limited to a 1K-caption, 10K-image LAION subset and a single A40 GPU timing setup.

Experimental setup

Datasets: LAION-5B subset (10K real images, 1K captions)

Benchmarks: LAION-5B subset (10K images, 1K captions)

Metrics: FID KID FIDc KIDc CLIP score latency (min)

Baselines: DemoFusion AccDiffusion FouriScale HiDiffusion BSRGAN

Builds on: DemoFusion FreeU HiDiffusion latent diffusion models attention swapping

Closest papers by meaning

• ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion (NeurIPS 2025 · similarity 0.91)
• Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs (NeurIPS 2024 · similarity 0.91)
• DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.91)
• Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.91)
• RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation (NeurIPS 2025 · similarity 0.91)
• Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.90)