Research map / Image generation / Fast diffusion sampling
Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the first exact framework for training diffusion models that sample the clean distribution from noisy-only training data, combining an Ambient Tweedie denoising score matching objective with a consistency loss, and applies it to fine-tune SDXL on noisy FFHQ/LAION latents.
Problem
Training diffusion models that provably sample the uncorrupted distribution when only noisy samples at a single noise level are available; prior approaches (SURE, Ambient Diffusion) only approximate the clean distribution or cannot reach noise levels below the data noise.
Approach
Apply Tweedie's formula twice (once w.r.t. clean X0 and once w.r.t. the noisy X_tn) to obtain a divergence-free ambient denoising score matching objective for noise levels above the data noise, and add a consistency loss to extend the denoiser to levels below the data noise, yielding the exact optimal denoiser for all noise levels.
Key results
- Models trained with consistency maintain FID comparable to the clean-data baseline for noise levels up to t_n=500 on FFHQ, and are better than models trained without consistency at all levels.
- Denoising MSE is comparable (e.g. latent MSE ~0.34 at eval t=900) across training noise levels 100, 500, and 800, supporting the claim that all models learn E[X0|Xt].
- Fine-tuning with noisy data reduces memorization: the t_n=500 model shows lower DINO similarity to training images than the clean-data model; SDXL memorization is found to be higher than previously reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the consistency loss during fine-tuning (ablation) | SDXL fine-tuned on FFHQ · FFHQ unconditional generation, training noise levels 100, 500, and 800 | FID | Helped |
| Add the consistency loss during fine-tuning (ablation) | SDXL fine-tuned on FFHQ · FFHQ unconditional generation, especially training noise levels 500 and 800 | generation image detail | Helped |
| Remove the consistency loss during fine-tuning (ablation) | SDXL fine-tuned on FFHQ · FFHQ unconditional generation as training corruption increases | FID | Hurt |
| Train on corrupted rather than clean data | SDXL fine-tuned on FFHQ · FFHQ training-data replication | DINO similarity to nearest training image | Helped |
| Vary the training noise level from 100 to 500 or 800 (ablation) | Ambient Tweedie restoration model · FFHQ restoration at eval noise levels 900, 800, 500, and 100 | Latent MSE: 0.3369 (0.0521); 0.3377 (0.0514); 0.3375 (0.0514); 0.2974 (0.0463); 0.2978 (0.0464); 0.3001 (0.0466); 0.2153 (0.0283); 0 | No clear effect |
| Vary the training noise level from 100 to 500 or 800 (ablation) | Ambient Tweedie restoration model · FFHQ restoration at eval noise levels 900, 800, 500, and 100 | Pixel MSE: 0.0802 (0.0257); 0.0804 (0.0253); 0.0796 (0.0254); 0.0566 (0.0219); 0.0566 (0.0222); 0.0570 (0.0220); 0.0219 (0.0092); 0 | No clear effect |
Surprising findings
- Extremely corrupted training images (noise level t=900) can be reconstructed almost pixel-perfectly by SDXL posterior sampling, suggesting more memorization than previously known.
- Models trained without consistency produce increasingly blurry generations as the training noise level increases, since they sample from the MMSE-denoised distribution.
Limitations
- Does not solve exact training from linearly corrupted data (e.g., inpainting/masking), unlike the consistency-based approach for noise corruption.
- Consistency loss increases training time by roughly 3x.
- Ambient Denoising Score Matching failed in preliminary experiments on very small datasets (<100 samples).
Experimental setup
Datasets: FFHQ LAION (10k subset) chest X-ray dataset
Benchmarks: FFHQ LAION
Metrics: FID MSE (latent and pixel) DINOv2 nearest-neighbor similarity
Baselines: Ambient Diffusion SURE-based diffusion training Noisier2Noise SDXL (clean-data fine-tuning)
Builds on: Ambient Diffusion Consistent Diffusion Models Noisier2Noise Denoising Score Matching
Closest papers by meaning
- The Emergence of Reproducibility and Consistency in Diffusion Models (ICML 2024 · similarity 0.92)
- Ambient Diffusion Omni: Training Good Models with Bad Data (NeurIPS 2025 · similarity 0.91)
- Elucidating the Exposure Bias in Diffusion Models (ICLR 2024 · similarity 0.90)
- Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion (ICML 2025 · similarity 0.90)
- Diffusion Models With Learned Adaptive Noise (NeurIPS 2024 · similarity 0.89)
- Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.