Research map / Image generation / Fast diffusion sampling
Immiscible Diffusion: Accelerating Diffusion Training with Noise Assignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Immiscible Diffusion, which assigns each image a noise sample within a mini-batch via linear assignment (quantized to fp16) before diffusing, speeding up training of Consistency Models, DDIM and Stable Diffusion.
Problem
Diffusion training is slow because each image can be diffused to any point in the noise space, so noise-data mappings are fully mixed and the denoising function is hard to optimize.
Approach
Batch-wise assignment of Gaussian noise to images that minimizes total image-noise L2 distance before diffusion, restricting each image's diffusible area while keeping the marginal noise Gaussian; implemented in one line with scipy linear_sum_assignment and accelerated by fp16 quantization of the distance computation.
Key results
- Immiscible Consistency Model reaches FID 10.00 threshold in 190k steps vs >800k for baseline on CIFAR-10, roughly 3x efficiency.
- Class-conditional Stable Diffusion on ImageNet reaches FID 16.43 at 20k steps, 1.49 lower than baseline.
- Batch-wise assignment reduces image-noise L2 distance by only about 2% and adds 22.8 ms at batch size 1024.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Assign batch noise to images using minimum-distance matching | DDIM · CIFAR-10, 20 inference steps | FID: 10.03 -> 7.46 | Helped |
| Assign batch noise to images using minimum-distance matching | DDIM · CIFAR-10, 1000, 500, 100, 50, and 20 inference steps | FID: 3.82 -> 3.67; 3.91 -> 3.74; 5.2 -> 4.32; 6.63 -> 5.14; 10.03 -> 7.46 | Helped |
| Assign batch noise to images using minimum-distance matching | Consistency Model · CIFAR-10, FID threshold 12.00 | training steps (k): 290 -> 110 | Helped |
| Assign batch noise to images using minimum-distance matching | Consistency Model · CIFAR-10, FID threshold 11.00 | training steps (k): 450 -> 140 | Helped |
| Assign batch noise to images using minimum-distance matching | Consistency Model · CIFAR-10, FID threshold 10.00 | training steps (k): >800 -> 190 | Helped |
| Assign batch noise to images using minimum-distance matching | Stable Diffusion · ImageNet class-conditional generation, 20k training steps | FID: 1.49 lower; 16.43 | Helped |
| Assign batch noise to images using minimum-distance matching | Stable Diffusion · ImageNet class-conditional generation, 20k training steps | CMMD: 1.436 -> 1.385 | Helped |
| Assign batch noise to images using minimum-distance matching | Stable Diffusion · ImageNet class-conditional generation, 20k training steps | CLIPScore: 28.55 -> 28.55 | No clear effect |
| Assign batch noise to images using minimum-distance matching | Stable Diffusion v1.4 fine-tuning · ImageNet, 5k training steps | FID: 11.45 -> 10.28 | Helped |
| Use L1 instead of L2 distance for assignment (ablation) | Immiscible DDIM · CIFAR-10, across 70.2k–163.8k training steps | FID: 5.28 vs 5.34; 4.56 vs 4.66; 4.13 vs 4.16; 3.81 vs 3.87; 3.70 vs 3.82 | Hurt |
| Remove optimal-transport matching while retaining immiscible image-noise assignment (ablation) | Immiscible Diffusion · Diffusion training, final stages | FID | No clear effect |
| Vary batch size while applying image-noise assignment (ablation) | Immiscible DDIM · DDIM, selected range of batch sizes | training efficiency | Helped |
Surprising findings
- Non-OT immiscible variant (assigning to flipped noise) achieves nearly the same gains as OT assignment, suggesting immiscibility rather than OT drives the improvement.
- Image-noise distance is reduced by only about 2% yet training speed improves substantially.
Limitations
- Assignment strategy is one straightforward approach and not necessarily optimal.
- Experiments limited to small-scale datasets; no validation on large-scale data such as LAION or text-to-image settings.
- Speedups on CelebA and tiny ImageNet are smaller (1.3x and 1.2x).
Experimental setup
Datasets: CIFAR-10 CelebA Tiny ImageNet ImageNet
Benchmarks: CIFAR-10 CelebA Tiny ImageNet ImageNet
Metrics: FID CMMD CLIPScore training steps to FID threshold assignment time
Baselines: Consistency Model DDIM Stable Diffusion non-OT immiscible variant
Builds on: Consistency Models DDIM Stable Diffusion flow matching with batch optimal transport
Closest papers by meaning
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.89)
- Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data (ICML 2024 · similarity 0.89)
- Diffusion Models With Learned Adaptive Noise (NeurIPS 2024 · similarity 0.89)
- Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy (NeurIPS 2024 · similarity 0.88)
- One-step Diffusion with Distribution Matching Distillation (CVPR 2024 · similarity 0.88)
- AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.