Research map / Image generation / Fast diffusion sampling
Noise Matters: Optimizing Matching Noise for Diffusion Classifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NoOp, which learns a dataset-specific noise plus an image-conditioned noise offset from a Meta-Network to replace random noise in diffusion classifiers, stabilizing and improving few-shot classification without costly noise ensembling.
Problem
Diffusion classifiers (DCs) suffer from noise instability: classification accuracy varies strongly with the randomly sampled noise, and existing fixes ensemble many noise samples, making inference slow.
Approach
Replace random noise in the diffusion classifier with a learned noise: a dataset-specific parameterized noise (frequency matching) plus an image-specific offset produced by a light U-Net Meta-Network (spatial matching), both optimized with a classification loss on z-score normalized distances.
Key results
- NoOp with only 2 shots on average beats the 5-noise ensemble DC across eight few-shot datasets, with 16-shot training taking about 24% of total time while gaining 3.19% over the 5-times ensemble.
- In cross-dataset transfer from 4-shot ImageNet, NoOp averages 37.65% vs 34.79% for 5x ensembling over eight target datasets (+2.86), winning on six of eight.
- Combining NoOp with TiF Learner improves FGVCAircraft 16-shot accuracy to 25.74% vs 21.03% for TiF alone and 14.61% for NoOp alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| optimize dataset-specific noise and image-specific noise offsets instead of using five sampled noises | Diffusion Classifier · ImageNet (source), 4-shot transfer | top-1 accuracy: 25.94 -> 26.34 | Helped |
| apply noise and Meta-Network optimized on 4-shot ImageNet instead of using five sampled noises | Diffusion Classifier · DTD (target), 4-shot ImageNet transfer | top-1 accuracy: +4.36 | Helped |
| apply noise and Meta-Network optimized on 4-shot ImageNet instead of using five sampled noises | Diffusion Classifier · CIFAR-10 (target), 4-shot ImageNet transfer | top-1 accuracy: -6.65 | Hurt |
| apply noise and Meta-Network optimized on 4-shot ImageNet instead of using five sampled noises | Diffusion Classifier · CIFAR-100 (target), 4-shot ImageNet transfer | top-1 accuracy: +11.40 | Helped |
| apply noise and Meta-Network optimized on 4-shot ImageNet instead of using five sampled noises | Diffusion Classifier · EuroSAT (target), 4-shot ImageNet transfer | top-1 accuracy: +0.94 | Helped |
| apply noise and Meta-Network optimized on 4-shot ImageNet instead of using five sampled noises | Diffusion Classifier · Flowers102 (target), 4-shot ImageNet transfer | top-1 accuracy: +12.01 | Helped |
| apply noise and Meta-Network optimized on 4-shot ImageNet instead of using five sampled noises | Diffusion Classifier · STL-10 (target), 4-shot ImageNet transfer | top-1 accuracy: -3.78 | Hurt |
| apply noise and Meta-Network optimized on 4-shot ImageNet instead of using five sampled noises | Diffusion Classifier · OxfordPets (target), 4-shot ImageNet transfer | top-1 accuracy: +3.49 | Helped |
| apply noise and Meta-Network optimized on 4-shot ImageNet instead of using five sampled noises | Diffusion Classifier · FGVCAircraft (target), 4-shot ImageNet transfer | top-1 accuracy: +1.14 | Helped |
| apply noise and Meta-Network optimized on 4-shot ImageNet instead of using five sampled noises | Diffusion Classifier · Average across target datasets, 4-shot ImageNet transfer | top-1 accuracy: +2.86 | Helped |
| combine NoOp with TiF Learner prompt optimization | Diffusion Classifier · ISIC-2019, 2-shot | top-1 accuracy: 20.80 -> 19.23 | Hurt |
| combine NoOp with TiF Learner prompt optimization | Diffusion Classifier · FGVCAircraft, 2-shot | top-1 accuracy: 12.51 -> 18.54 | Helped |
Surprising findings
- Prompt optimization (TiF Learner) can reduce performance below zero-shot DC at 1 shot on six datasets, while noise optimization is more stable.
- Optimized noises have statistics (mean, variance, Gaussian pdf) that differ only slightly from random noise yet lie in low-probability regions of the standard Gaussian.
Limitations
- Inference still requires one forward pass per candidate category, so it remains costlier than CLIP-style classifiers.
- Results on ImageNet transfer are mixed: some target datasets (e.g., CIFAR-10, STL-10) are worse than ensembling.
- Timestep is fixed at t=500 in main experiments; the tradeoff across timesteps is only partly explored.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Flowers102 DTD OxfordPets EuroSAT STL-10 FGVCAircraft ImageNet ISIC-2019
Benchmarks: CIFAR-10 CIFAR-100 Flowers102 DTD OxfordPets EuroSAT STL-10 FGVCAircraft ISIC-2019 ImageNet
Metrics: top-1 accuracy CLIP Score high-frequency signal ratio training time and inference time
Baselines: Zero-shot DC Ensemble DC (5 noises) TiF Learner Textual inversion prompt learning DC
Builds on: Diffusion Classifier TiF Learner CLIP few-shot protocol Textual inversion
Closest papers by meaning
- CleanDIFT: Diffusion Features without Noise (CVPR 2025 · similarity 0.87)
- Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training (NeurIPS 2024 · similarity 0.86)
- Label-Noise Robust Diffusion Models (ICLR 2024 · similarity 0.86)
- Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR 2024 · similarity 0.86)
- Directional Label Diffusion Model for Learning from Noisy Labels (CVPR 2025 · similarity 0.85)
- Classification Diffusion Models: Revitalizing Density Ratio Estimation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.