Research map / Image generation / Editing and style transfer
Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Makes latent diffusion models fractionally shift-equivariant by combining alias-free VAE/U-Net modules, an equivariance loss, and equivariant (cross-frame style) attention, improving consistency in video editing and image-to-image tasks.
Problem
Latent diffusion models produce inconsistent outputs under small pixel shifts or warps of the input noise, because aliasing amplifies during VAE training and multi-step U-Net inference and self-attention is not shift-equivariant.
Approach
Adapts StyleGAN3/AFC-style anti-aliasing to the VAE and U-Net of Stable Diffusion, adds an equivariance loss over fractional (Fourier) shifts during training, and uses reference-frame keys and values (equivariant attention, identical in form to cross-frame attention) in both training and inference.
Key results
- AF-VAE raises encoder SPSNR from 25.07 (SD VAE) to 45.10 and decoder SPSNR from 22.31 to 33.88 on ImageNet 256x256, with reconstruction PSNR 25.53 vs 26.94.
- On FFHQ unconditional LDM, AF-LDM raises latent SPSNR from 27.47 (LDM with SD VAE) to 40.94 but FID rises from 14.91 to 19.05.
- On Sintel warping, AF-SD inversion warping PSNR is 26.10 vs 17.53 for SD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ideal sampling, filtered nonlinearity, and equivariance loss | SD VAE† · ImageNet 256×256 validation | encoder SPSNR: 19.71 -> 45.10 | Helped |
| Add ideal sampling, filtered nonlinearity, and equivariance loss | SD VAE† · ImageNet 256×256 validation | reconstruction PSNR: 26.45 -> 25.53 | Hurt |
| Add ideal sampling (ablation) | SD VAE† · ImageNet 256×256 validation | encoder SPSNR: 19.71 -> 21.51 | Helped |
| Add filtered nonlinearity after ideal sampling (ablation) | SD VAE† · ImageNet 256×256 validation | encoder SPSNR: 21.51 -> 29.49 | Helped |
| Add equivariance loss without alias-free modules (ablation) | SD VAE† · ImageNet 256×256 validation | reconstruction PSNR: 26.45 -> 24.59 | Hurt |
| Add equivariance loss without alias-free modules (ablation) | SD VAE† · ImageNet 256×256 validation | encoder SPSNR: 19.71 -> 41.46 | Helped |
| Replace SD VAE with AF-VAE (ablation) | LDM · FFHQ 256×256 | image SPSNR: 12.38 -> 12.18 | No clear effect |
| Enable CFA in inference (ablation) | LDM w/ SD VAE · FFHQ 256×256 | image SPSNR: 12.38 -> 21.69 | Helped |
| Add ideal sampling (ablation) | LDM w/ AF-VAE · FFHQ 256×256 | FID: 15.27 -> 17.03 | Hurt |
| Add filtered nonlinearity (ablation) | LDM w/ AF-VAE + ideal sampling · FFHQ 256×256 | FID: 17.03 -> 14.28 | Helped |
| Add equivariance loss without CFA (ablation) | LDM w/ AF-VAE · FFHQ 256×256 | image SPSNR: 14.15 -> 17.65 | Helped |
| Use CFA in equivariance loss (ablation) | LDM w/ AF-VAE + equivariance loss · FFHQ 256×256 | image SPSNR: 17.65 -> 28.06 | Helped |
Surprising findings
- Cross-frame attention is needed during training as well as inference to suppress aliasing; without it the equivariance loss mainly fixes attention rather than aliasing.
- Anti-aliasing modules alone can worsen shift-equivariance as training progresses, with aliasing amplified during VAE training.
Limitations
- Equivariance loss and alias-free modules reduce generation quality (FID rises from 14.91 to 19.05 on FFHQ; to 39.14 when equivariance loss is added without CFA).
- Shift-equivariance is studied mainly for pixel-level Fourier shifts; warping robustness is evaluated only on a limited set (Sintel).
Experimental setup
Datasets: ImageNet FFHQ LAION Aesthetic 6.5+ Sintel
Benchmarks: ImageNet validation FFHQ Sintel
Metrics: PSNR rFID FID shift PSNR (SPSNR) warping PSNR
Baselines: Stable Diffusion VAE SD latent I2SB YOSO Warped Diffusion
Builds on: Stable Diffusion StyleGAN3 Alias-Free Convnets I2SB YOSO
Closest papers by meaning
- Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models (CVPR 2024 · similarity 0.89)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.88)
- Improving the Diffusability of Autoencoders (ICML 2025 · similarity 0.88)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.88)
- Gradient-free Decoder Inversion in Latent Diffusion Models (NeurIPS 2024 · similarity 0.88)
- DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.