Research map / Image generation / Fast diffusion sampling
Three Things We Need to Know About Transferring Stable Diffusion to Visual Dense Prediciton Tasks
Manyuan Zhang, Guanglu Song, Xiaoyu Shi, Yu Liu, Hongsheng Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Stable Diffusion UNet as backbone instead of CLIP-pretrained ViT-B (DenseCLIP) | Semantic FPN head · ADE20K val | mIoU: 50.6 -> 54.1 | Improved |
| add small-scale noise (t=10) before feature extraction (ablation) | SD-UNet feature encoder · ADE20K val | mIoU: 53.2 -> 54.1 | Improved |
| add noise rather than zero noise (ablation) | SD-UNet feature encoder · ADE20K val | mIoU | Improved |
| concatenate cross-attention maps from down and up stages with UNet features (ablation) | SD-UNet feature encoder · ADE20K val | mIoU: 52.4 -> 54.1 | Improved |
| use later Stable Diffusion checkpoint with better generation ability (v1.1 to v1.5) (ablation) | SD-UNet feature encoder · ADE20K val | mIoU: 49.7 -> 54.1 | Improved |
| provide class names as text input instead of none (ablation) | SD-UNet feature encoder · ADE20K val | mIoU: 49.5 -> 54.1 | Improved |
| use BLIP2 captions as text input instead of all class names (ablation) | SD-UNet feature encoder · ADE20K val | mIoU: 54.1 -> 55.0 | Improved |
| tune only the text encoder with UNet frozen (ablation) | Stable Diffusion · ADE20K val | mIoU: 38.3 -> 48.4 | Improved |
| replace full tuning with LoRA on UNet (ablation) | SD-UNet · ADE20K val | mIoU: 54.1 -> 51.3 | Hurt |
| use searched protocol (LoRA in cross-attention + adapter after FFN) instead of LoRA everywhere (ablation) | SD-UNet · ADE20K val | mIoU: 51.3 -> 52.1 | Improved |
| provide ground-truth classes as text input (oracle) (ablation) | SD-UNet feature encoder · ADE20K val | mIoU: 54.1 -> 71.5 | Improved |
| apply method to NYUv2 depth estimation vs VPD | SD-UNet · NYUv2 depth | RMSE: 0.254 -> 0.241 | Improved |
Closest papers by meaning
- What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.91)
- Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction (ICLR 2025 · similarity 0.90)
- Exploiting Diffusion Prior for Generalizable Dense Prediction (CVPR 2024 · similarity 0.89)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.88)
- Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations (NeurIPS 2025 · similarity 0.88)
- Scaling Properties of Diffusion Models For Perceptual Tasks (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
