AI Paper Atlas

Research map / Image generation / Fast diffusion sampling

Three Things We Need to Know About Transferring Stable Diffusion to Visual Dense Prediciton Tasks

Manyuan Zhang, Guanglu Song, Xiaoyu Shi, Yu Liu, Hongsheng Li

ECCV 2024 · Image generation › Fast diffusion sampling · approach: Diffusion models as perception tools

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use Stable Diffusion UNet as backbone instead of CLIP-pretrained ViT-B (DenseCLIP)Semantic FPN head · ADE20K valmIoU: 50.6 -> 54.1Improved
add small-scale noise (t=10) before feature extraction (ablation)SD-UNet feature encoder · ADE20K valmIoU: 53.2 -> 54.1Improved
add noise rather than zero noise (ablation)SD-UNet feature encoder · ADE20K valmIoUImproved
concatenate cross-attention maps from down and up stages with UNet features (ablation)SD-UNet feature encoder · ADE20K valmIoU: 52.4 -> 54.1Improved
use later Stable Diffusion checkpoint with better generation ability (v1.1 to v1.5) (ablation)SD-UNet feature encoder · ADE20K valmIoU: 49.7 -> 54.1Improved
provide class names as text input instead of none (ablation)SD-UNet feature encoder · ADE20K valmIoU: 49.5 -> 54.1Improved
use BLIP2 captions as text input instead of all class names (ablation)SD-UNet feature encoder · ADE20K valmIoU: 54.1 -> 55.0Improved
tune only the text encoder with UNet frozen (ablation)Stable Diffusion · ADE20K valmIoU: 38.3 -> 48.4Improved
replace full tuning with LoRA on UNet (ablation)SD-UNet · ADE20K valmIoU: 54.1 -> 51.3Hurt
use searched protocol (LoRA in cross-attention + adapter after FFN) instead of LoRA everywhere (ablation)SD-UNet · ADE20K valmIoU: 51.3 -> 52.1Improved
provide ground-truth classes as text input (oracle) (ablation)SD-UNet feature encoder · ADE20K valmIoU: 54.1 -> 71.5Improved
apply method to NYUv2 depth estimation vs VPDSD-UNet · NYUv2 depthRMSE: 0.254 -> 0.241Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy