Research map / Image generation / Text-to-image diffusion
DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Repurposes a pretrained SD3 text-to-image diffusion transformer into a single generalist perception model (depth, normal, keypoints, interactive/entity/instance segmentation) by unifying outputs in RGB space and concatenating image, noise, task-prompt and point tokens.
Problem
Building a generalist visual perception model across multiple tasks with limited data and compute, without training from scratch or relying on task-specific modules.
Approach
Treat each perception task as RGB-space image generation conditioned on the input image via token-wise concatenation in a pretrained SD3 DiT, keeping the architecture nearly unchanged so the generative prior is preserved; point prompts are encoded by a small MLP with learned valid/invalid embeddings, and mild classifier-free guidance is used for depth and normal.
Key results
- On zero-shot depth, Ours reaches AbsRel 0.061 on NYUv2 and 0.050 on ETH3D, competitive with specialist models such as Depth Pro and Metric3D v2.
- Interactive segmentation mIoU of 47.10 with 1 point is reported as on par with SAM-vit-h (48.90) using 600K vs 1B training images.
- Few-step inference: depth AbsRel on NYUv2 degrades only from 0.061 (28-step) to 0.072 (1-step).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 5-point rather than 1-point prompts (ablation) | DICEPTION interactive segmentation · 23 validation datasets | mIoU: 47.1 -> 57.2 | Helped |
| Set classifier-free guidance to 2 rather than 1 (ablation) | DICEPTION depth estimation · KITTI | AbsRel: 0.075 -> 0.069 | Helped |
| Increase classifier-free guidance from 2 to 3 (ablation) | DICEPTION depth estimation · KITTI | AbsRel: 0.069 -> 0.092 | Hurt |
| Increase classifier-free guidance from 2 to 3 (ablation) | DICEPTION normal estimation · NYUv2 | mean angular error: 18.338 -> 19.817 | Hurt |
| Vary classifier-free guidance (ablation) | DICEPTION interactive segmentation · 23 validation datasets | mIoU: 47.10; 47.12; 47.08; 46.91; 46.57 | No clear effect |
| Replace SD3 with PixArt-alpha-600M (ablation) | DICEPTION depth estimation · KITTI | AbsRel: 0.069 -> 0.093 | Hurt |
| Reduce inference from 28 steps to 1 step (ablation) | DICEPTION interactive segmentation · 23 validation datasets | mIoU: 47.10 -> 42.53 | Hurt |
| Use token-wise rather than channel-wise input concatenation (ablation) | DICEPTION · Depth and surface normal estimation | training convergence speed | Helped |
| Use a UNet-based SDXL architecture rather than the SD3 DiT architecture (ablation) | DICEPTION multi-task perception · Depth and surface normal estimation | task performance | Hurt |
| Train with one-step rather than multi-step denoising (ablation) | DICEPTION multi-task perception · Multi-task perception | failure cases | Hurt |
| Train a single-task model rather than the multi-task model (ablation) | DICEPTION · Depth and surface normal estimation | performance | No clear effect |
| Train with pixel-aligned perception targets (ablation) | DICEPTION · Low-level image processing | fine-detail preservation | Helped |
Surprising findings
- Multi-task 1-step training yields more failure cases than single-task, attributed to overlapping denoising trajectories across tasks.
- Channel-wise concatenation and UNet-based (SDXL) variants fail on multi-task learning, while token-wise DiT concatenation works.
Limitations
- Pose and instance segmentation metrics are low (pose AP 57.8 vs 76-78 for specialists; instance AP 33.2 vs 50.1 for Mask2Former), partly attributed to post-processing.
- Multi-task model shows no clear gain over single-task models at the current ~1.8M sample scale.
- Underperforms SAM on some out-of-distribution datasets such as WoodScape fisheye.
Experimental setup
Datasets: OpenImages SA-1B AM2k AIM500 P3M-10k EntityV2 COCO-Rem MS COCO 2017 Chest X-Ray LOL-v2 Kaggle Brain Tumor NYUv2
Benchmarks: NYUv2 KITTI ScanNet DIODE ETH3D MS COCO 2017 23 interactive segmentation validation datasets WoodScape
Metrics: AbsRel delta1 mean/median angular error 11.25/22.5/30 degree accuracy mIoU AP AR
Baselines: Depth Anything Depth Pro Metric3D v2 Marigold GeoWizard Painter Unified-IO 4M-XL OneDiffusion StableNormal SAM-vit-h Mask2Former
Builds on: Stable Diffusion 3 flow matching Painter Marigold LoRA
Closest papers by meaning
- DEEM: Diffusion models serve as the eyes of large language models for image perception (ICLR 2025 · similarity 0.90)
- What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.89)
- Scaling Properties of Diffusion Models For Perceptual Tasks (CVPR 2025 · similarity 0.89)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.87)
- Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception (ICLR 2025 · similarity 0.87)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.