Research map / Image generation / Fast diffusion sampling
Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits feature selection for diffusion features by enlarging the activation pool (incl. q/k and embedded ViT activations), filtering qualitatively via three universal properties, and proposing selected feature sets for SDv1.5 and SDXL evaluated on correspondence and segmentation.
Problem
Selecting a small, effective subset of diffusion U-Net activations as dense features for discriminative tasks remains unresolved, since prior work only considered inter-module activations and a full quantitative sweep over the expanded candidate pool is infeasible.
Approach
Identify three universal properties of diffusion U-Nets (asymmetric diffusion noises across the U, in-resolution granularity changes, and locality in self-attention without positional embeddings) to qualitatively filter the candidate activations before a reduced quantitative comparison, then select off-the-shelf feature combinations mainly from embedded ViT modules.
Key results
- Ours-XL-t reaches 83.90 PCK@0.1_img on SPair-71k versus 72.56 for DHF.
- Ours-XL-t achieves 45.71 mIoU on ADE20K versus 40.89 for Meta Prompts.
- Ours-XL-t reaches 66.3 mIoU on label-scarce Horse-21, slightly above DDPM at 65.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select broader SDv1.5 activations, including ViT features, instead of conventional inter-module features | SDv1.5 diffusion features · SPair-71k | PCK@0.1_img: 75.14 -> 77.78 | Helped |
| Select broader SDXL activations, including embedded ViT features, instead of conventional inter-module features | SDXL diffusion features · SPair-71k | PCK@0.1_img: 66.00 -> 81.72 | Helped |
| Add attention maps and feature amalgamation to selected features | Ours-XL-t · SPair-71k | PCK@0.1_img: 72.56 -> 83.90 | Helped |
| Use selected SDXL features with additional lightweight techniques rather than Meta Prompts | Semantic segmentation · ADE20K | mIoU: 40.89 -> 45.71 | Helped |
| Use selected SDXL features with additional lightweight techniques rather than Meta Prompts | Semantic segmentation · CityScapes | mIoU: 71.94 -> 75.89 | Helped |
| Use selected SDXL, SDv1.5, and Playground v2 features with additional lightweight techniques instead of DDPM | Label-scarce segmentation · Horse-21 | mIoU: 65.0 -> 66.3 | Helped |
| Choose features using the generic solution rather than a solution selected only on simple scenes | SDXL feature selection · Horse-21 simple scene and Bedroom-28 complex scene | mIoU: 63.70 -> 63.34; 45.41 -> 47.55 | Mixed |
Surprising findings
- Legacy SDXL features (inter-module baseline) underperform SDv1.5 on semantic correspondence (66.00 vs 75.14 PCK), despite SDXL being the more advanced model.
- On label-scarce segmentation, the DDPM baseline trained per dataset in image space outperforms most diffusion-feature variants, including the proposed ones without extra techniques.
Limitations
- Properties are verified visually and mainly on SDXL; universality to DiT models is not established and is left as future work.
- The best results rely on extra techniques (attention maps, feature amalgamation across models), and the baseline DDPM approach remains competitive on label-scarce segmentation.
Experimental setup
Datasets: SPair-71k ADE20K CityScapes Horse-21 (LSUN)
Benchmarks: SPair-71k ADE20K CityScapes Horse-21
Metrics: PCK@0.1 img PCK@0.1 bbox mIoU
Baselines: DINO DHPF DIFT DHF MaskCLIP ODISE VPD Meta Prompts DDPM MAE SwAV DatasetDDPM
Builds on: DIFT DHF DDPM segmentation (Baranchuk et al.) Meta Prompts VPD
Closest papers by meaning
- Diffusion Classifiers Understand Compositionality, but Conditions Apply (NeurIPS 2025 · similarity 0.90)
- Why DiT Models Underperform as Representation Learners without Long Skip Connections (NeurIPS 2026 · similarity 0.90)
- An Efficient Framework for Enhancing Discriminative Models via Diffusion Techniques (AAAI 2025 · similarity 0.89)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.87)
- CleanDIFT: Diffusion Features without Noise (CVPR 2025 · similarity 0.87)
- FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.