Research map / Image generation / Text-to-image diffusion
What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Investigates design choices for repurposing text-to-image diffusion models to dense perception and proposes GenPercept, a deterministic one-step fine-tuning paradigm using the diffusion UNet with optional decoders and pixel-wise losses.
Problem
Design choices (multi-step diffusion mechanism, training strategy, inference ensembling, fine-tuning data quality) for transferring T2I diffusion priors to dense perception tasks lack systematic justification.
Approach
Show that the multi-step diffusion process can be reduced to deterministic one-step fine-tuning via scheduler hyperparameters without performance loss, and that high-quality fine-tuning data and task-specific pixel-wise supervision matter; package these into GenPercept, which fine-tunes the Stable Diffusion UNet with a one-step inference pipeline.
Key results
- One-step GenPercept matches multi-step diffusion performance on dense tasks while offering much faster inference than multi-step Marigold-style methods.
- High-quality synthetic fine-tuning data is shown to be paramount for both semantic and geometry perception tasks.
- Task-specific pixel-wise supervision with customized decoders improves fine-grained details beyond latent-space-only supervision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use high-quality fine-tuning data | diffusion-model fine-tuning for dense perception · dense visual perception tasks | performance | Helped |
| replace multi-step generation with one-step fine-tuning | GenPercept · dense visual perception tasks | performance | No clear effect |
| replace multi-step inference with single-step inference | diffusion models for dense perception · dense visual perception tasks | inference speed | Helped |
| add task-specific supervision | diffusion-model fine-tuning · dense visual perception tasks | fine-grained detail | Helped |
Surprising findings
- Strict adherence to the traditional multi-step diffusion process is unnecessary; single-step inference performs similarly.
- Multi-step stochastic generation and inference ensembling are not required for strong performance.
Limitations
- Scaling of fine-tuning data volume not explored
- Unclear whether fine-grained details come from LAION-scale pretraining or the diffusion paradigm itself
Experimental setup
Datasets: Hypersim Virtual KITTI synthetic fine-tuning datasets real-world datasets
Benchmarks: NYUv2 KITTI ScanNet ETH3D DIS matting benchmarks
Metrics: AbsRel delta1 mean angular error F-score MAE
Baselines: Marigold GeoWizard DepthFM DINOv2
Builds on: Marigold Stable Diffusion LoRA
Closest papers by meaning
- Scaling Properties of Diffusion Models For Perceptual Tasks (CVPR 2025 · similarity 0.92)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.92)
- DEEM: Diffusion models serve as the eyes of large language models for image perception (ICLR 2025 · similarity 0.91)
- Exploiting Diffusion Prior for Generalizable Dense Prediction (CVPR 2024 · similarity 0.91)
- Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction (ICLR 2025 · similarity 0.90)
- Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.