Research map / Image generation / Image restoration
Residual Learning in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a residual learning framework that trains a parameterized correction function to map intermediate diffusion-model samples to clean images, rectifying the sampling trajectory of pre-trained diffusion models and transferring across DMs on the same dataset.
Problem
Diffusion model sampling trajectories deviate from the true trajectory due to score estimation and discretization errors, leaving residual artifacts that reduce image quality.
Approach
Train a correction function f(x_i^ODE, i) -> x_0 on reverse-ODE-simulated residual pairs from real images, then apply it at an intermediate step during sampling of any pre-trained DM on the same dataset; the correction is plug-and-play and transferable across DMs, with a lightweight distilled variant.
Key results
- On CIFAR-10, the correction improves EDM from FID 2.04 to 1.93 and NSCNv2 from 10.87 to 9.89.
- On ImageNet 512x512, the correction improves ADM FID from 23.24 to 19.97 and DiT from 12.03 to 11.37.
- A correction function trained on EDM transfers to other CIFAR-10 DMs, e.g. DDPM FID 3.17 to 3.06 and NSCNv2 10.87 to 10.01 with EDM-H.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add residual correction to pretrained diffusion models | Diffusion models on ImageNet 256×256 and 512×512 · ImageNet | FID: 10.94→9.86; 23.24→19.97 | Helped |
| Add residual correction to pretrained diffusion models | Diffusion models on CIFAR-10 · CIFAR-10 | FID: 10.87→9.89; 3.17→3.05; 2.55→2.44; 2.20→2.15; 2.04→1.93 | Helped |
| Add residual correction to pretrained diffusion models | Diffusion models on AFHQv2 and CelebA · AFHQv2 64×64 and CelebA 64×64 | FID: 2.16→2.10; 1.96→1.91; 2.39→2.30; 47.30→42.10; 2.32→2.22; 1.90→1.85; 1.85→1.80; 1.75→1.71 | Helped |
| Increase correction-function size from EDM-T to EDM-H (ablation) | Residual correction + EDM on CIFAR-10 · CIFAR-10 | FID: 2.03→1.93 | Helped |
| Increase correction-function size from EDM-T to EDM-H (ablation) | Residual correction + EDM on CIFAR-10 · CIFAR-10 | IS: 9.84→9.91 | Helped |
| Use L2 rather than other metric functions to train the correction (ablation) | Residual correction on CIFAR-10 · CIFAR-10 | FID | Helped |
| Randomly choose the number of PF ODE reverse steps (ablation) | Residual correction on CIFAR-10 · CIFAR-10 | FID | Helped |
| Use SNR weighting for the correction loss (ablation) | Residual correction on CIFAR-10 · CIFAR-10 | FID | Helped |
| Combine masking, Gaussian noise, rotation, and cropping augmentations (ablation) | Residual correction on CIFAR-10 · CIFAR-10 | FID | Helped |
| Distill a heavy correction function into a lightweight correction function (ablation) | Residual correction on CIFAR-10 · CIFAR-10 | FID: 2.03→2.00; 2.03→1.99; 1.99→1.97; 1.99→1.96; 1.95→1.93; 1.95→1.93 | Helped |
| Add transferable residual correction | Training-free samplers on CIFAR-10 · CIFAR-10 | FID: 5.28→5.12; 3.02→2.94; 2.69→2.65; 6.03→5.89; 2.75→2.69; 2.65→2.62; 4.17→4.11; 2.86→2.82; 2.57→2.55; 20.89→19.94; 16.59→16 | Helped |
Surprising findings
- Correction functions trained on one DM improve other DMs trained on the same dataset, suggesting shared sampling error patterns.
- Lightweight correction functions (4-64x smaller) retain most of the gain.
Limitations
- Gains are modest (often a few hundredths of FID on CIFAR-10).
- Correction function must be trained per dataset and transfers only across DMs on the same dataset.
- Additional training cost of the correction network is not quantified in the text provided.
Experimental setup
Datasets: CIFAR-10 CelebA 64x64 AFHQv2 64x64 ImageNet 256x256 ImageNet 512x512
Benchmarks: CIFAR-10 CelebA 64x64 AFHQv2 64x64 ImageNet 256x256 ImageNet 512x512
Metrics: FID IS
Baselines: ADM ADM-G LDM-4 LDM-8 DiT NSCNv2 DDPM SDE (VE) EDM DPM-Solver DEIS
Builds on: score-based SDE EDM DPM-Solver DEIS
Closest papers by meaning
- Learning Diffusion Priors from Observations by Expectation Maximization (NeurIPS 2024 · similarity 0.86)
- Diffusion Models With Learned Adaptive Noise (NeurIPS 2024 · similarity 0.85)
- Neural Diffusion Models (ICML 2024 · similarity 0.84)
- A Mixture-Based Framework for Guiding Diffusion Models (ICML 2025 · similarity 0.84)
- Improving Probabilistic Diffusion Models With Optimal Diagonal Covariance Matching (ICLR 2025 · similarity 0.84)
- Probing the Latent Hierarchical Structure of Data via Diffusion Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.