Research map / Image generation / Fast diffusion sampling
CleanDIFT: Diffusion Features without Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a copy of Stable Diffusion to produce noise-free, timestep-independent semantic features from clean images by aligning them to noisy-timestep diffusion features.
Problem
Diffusion features for downstream tasks require adding noise to images and tuning a timestep per task, which degrades information and needs costly ensembling.
Approach
Treat the diffusion model as a family of timestep-dependent feature extractors and distill them into one trainable copy that takes clean images with no timestep input, using timestep-conditioned projection heads and cosine alignment at 11 U-Net feature maps.
Key results
- On SPair-71k, CleanDIFT improves DIFT from 66.53 to 68.32 PCK@0.1 (img) and Telling Left from Right from 77.07 to 78.40.
- Outperforms the single-step version of Luo et al. by 9.0 PCK points (img) while being about 50x faster than their DDIM-inversion method (which reaches 72.75).
- Depth estimation RMSE on NYUv2 improves from 0.469 (SD 2.1 DIFT-like) to 0.444 with a linear probe.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard diffusion features with CleanDIFT features in DIFT | DIFT · SPair-71k test | PCKimg (ω=0.1): 66.53 -> 68.32; +1.79 | Helped |
| Replace standard diffusion features with CleanDIFT features in DIFT | DIFT · SPair-71k test | PCKbbox (ω=0.1): 59.57 -> 61.43; +1.86 | Helped |
| Replace standard diffusion features with CleanDIFT features | A Tale of Two Features · SPair-71k test | PCKimg (ω=0.1): 72.31 -> 73.35; +1.04 | Helped |
| Replace standard diffusion features with CleanDIFT features | A Tale of Two Features · SPair-71k test | PCKbbox (ω=0.1): 63.73 -> 64.81; +1.08 | Helped |
| Replace standard diffusion features with CleanDIFT features | Telling Left from Right · SPair-71k test | PCKimg (ω=0.1): 77.07 -> 78.40; +1.33 | Helped |
| Replace standard diffusion features with CleanDIFT features | Telling Left from Right · SPair-71k test | PCKbbox (ω=0.1): 68.64 -> 69.99; +1.35 | Helped |
| Use CleanDIFT features instead of standard noisy diffusion features | Linear depth probe · NYUv2 | RMSE: 0.469 -> 0.444 | Helped |
| Reuse probes trained on noisy features for CleanDIFT features | Depth estimation probe · NYUv2 | RMSE: 0.469 -> 0.453 | Helped |
| Use cosine similarity rather than L1 or L2 alignment (ablation) | CleanDIFT · SPair-71k test subset | PCKimg (ω=0.1): 66.23; 66.49; 66.91 -> 68.32 | Helped |
| Use cosine similarity rather than L1 or L2 alignment (ablation) | CleanDIFT · SPair-71k test subset | PCKbbox (ω=0.1): 59.13; 59.43; 60.00 -> 61.43 | Helped |
| Add projection heads during alignment (ablation) | CleanDIFT with cosine similarity · SPair-71k test subset | PCKimg (ω=0.1): 68.16 -> 68.32 | Helped |
| Add projection heads during alignment (ablation) | CleanDIFT with cosine similarity · SPair-71k test subset | PCKbbox (ω=0.1): 61.29 -> 61.43 | Helped |
Surprising findings
- Simply feeding the clean image with a non-zero timestep to the diffusion model does not improve features; performance degrades faster than with noisy input.
- Noise ensembling (averaging over 8 noise samples) does not remove the benefit; CleanDIFT gains are larger without averaging.
Limitations
- Gains on semantic correspondence over the best prior method are modest (about 1.3 PCK points) when combined with Telling Left from Right.
- Projection heads add about 45M trainable parameters for SD 2.1 and give only small gains (0.24 PCK points) in the ablation.
Experimental setup
Datasets: COYO-700M
Benchmarks: SPair-71k NYUv2 PASCAL VOC ImageNet-1k
Metrics: PCK RMSE mIoU kNN accuracy
Baselines: DIFT A Tale of Two Features Telling Left from Right DINOv2+NN Luo et al. DDIM inversion method OpenCLIP MAE DINO iBOT DINOv2
Builds on: DIFT A Tale of Two Features Telling Left from Right Stable Diffusion
Closest papers by meaning
- Unsupervised Region-Based Image Editing of Denoising Diffusion Models (AAAI 2025 · similarity 0.90)
- Interpretable Diffusion via Information Decomposition (ICLR 2024 · similarity 0.89)
- Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data (ICML 2024 · similarity 0.88)
- NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models (CVPR 2024 · similarity 0.88)
- Why DiT Models Underperform as Representation Learners without Long Skip Connections (NeurIPS 2026 · similarity 0.88)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.