Research map / Image generation / Text-to-image diffusion
Exploiting Diffusion Prior for Generalizable Dense Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a pre-trained latent T2I diffusion model (Stable Diffusion) into a deterministic image-to-prediction mapper via an interpolation-based diffusion process, fine-tuned with LoRA on 10K bedroom images for generalizable dense prediction.
Problem
Off-the-shelf dense predictors (depth, normals, segmentation, intrinsic decomposition) fail on T2I-generated, highly imaginative images; T2I diffusion models are stochastic while dense prediction is deterministic, and fine-tuning risks losing generalizability.
Approach
Reformulate the diffusion forward process as interpolation between input RGB image and output prediction (weight on x grows over time), giving a deterministic image-to-prediction mapping; use v-prediction and LoRA on attention layers to retain the T2I prior.
Key results
- On 3D property estimation (out-of-domain), DMP achieves normal L1 0.0872 and angular error 0.1886, versus SPADE 0.1350/0.3006 among baselines.
- On semantic segmentation (in-domain), DMP reaches mean mIoU 0.6127 versus 0.4276 for the best GAN baseline SPADE.
- Intrinsic decomposition MSE for DMP is an order of magnitude lower than most baselines (albedo in-domain 0.0041 vs SPADE 0.0021 being the exception on albedo; SPADE is comparable).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DMP instead of SPADE | dense prediction · Surface normals, in-domain | L1: 0.0708 -> 0.0514 | Helped |
| Use DMP instead of SPADE | dense prediction · Surface normals, out-of-domain | Ang: 0.2833 -> 0.1886 | Helped |
| Use DMP instead of SPADE | dense prediction · Depth, in-domain | REL: 0.2132 -> 0.1072 | Helped |
| Use DMP instead of SPADE | dense prediction · Depth, out-of-domain | RMSE: 0.2554 -> 0.1360 | Helped |
| Use DMP instead of SPADE | semantic segmentation · In-domain mean | mIoU: 0.4276 -> 0.6127 | Helped |
| Use DMP instead of SPADE | intrinsic image decomposition · Albedo, in-domain | MSE: 0.0021 -> 0.0041 | Hurt |
| Use DMP instead of SPADE | intrinsic image decomposition · Shading, out-of-domain | MSE: 0.0040 -> 0.0070 | Hurt |
| Predict the input image x instead of using v-prediction (ablation) | DMP · Surface normals, in-domain | L1: 0.0514 -> 0.0736 | Hurt |
| Predict the input image x instead of using v-prediction (ablation) | DMP · Surface normals, out-of-domain | L1: 0.0872 -> 0.1319 | Hurt |
| Predict the output y instead of using v-prediction (ablation) | DMP · Surface normals, in-domain | L1: 0.0514 -> 0.0590 | Hurt |
| Predict the output y instead of using v-prediction (ablation) | DMP · Surface normals, out-of-domain | L1: 0.0872 -> 0.0888 | Hurt |
| Use a single diffusion step instead of 5 steps (ablation) | DMP · Surface normal prediction, arbitrary images | generalization | Hurt |
Surprising findings
- Predicting the input image x or the output y directly, or using a single diffusion step, generalizes poorly to unseen images, while v-prediction with 5 steps generalizes best.
- The GAN-based SPADE baseline is competitive with or better than several diffusion-based baselines on intrinsic decomposition.
Limitations
- Training labels are pseudo-ground-truth from off-the-shelf predictors, so performance is bounded by those teachers' errors.
- Evaluation relies on synthetic images and pseudo labels rather than real benchmark ground truth.
- Semantic segmentation is handled via color-map regression with nearest-color conversion, which is an indirect formulation.
Experimental setup
Datasets: synthetic bedroom images generated by Stable Diffusion with LLM-generated prompts SUN indoor category names as out-of-domain keywords
Benchmarks: in-domain synthetic bedroom test set out-of-domain synthetic SUN-keyword test set
Metrics: L1 angular error REL delta threshold accuracy RMSE mIoU pixel accuracy MSE
Baselines: SPADE DRIT++ SDEdit DDIB InstructPix2Pix (hard) InstructPix2Pix (learned) VISII
Builds on: latent diffusion (Stable Diffusion) LoRA v-prediction blending-based diffusion (Bansal et al.) DDIB SDEdit
Closest papers by meaning
- Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction (ICLR 2025 · similarity 0.92)
- What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.91)
- Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models (CVPR 2024 · similarity 0.89)
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.88)
- Learning Diffusion Texture Priors for Image Restoration (CVPR 2024 · similarity 0.88)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.