Research map / Image generation / Fast diffusion sampling
Scaling Properties of Diffusion Models For Perceptual Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that diffusion models (DiT) fine-tuned for perception tasks (depth, optical flow, amodal segmentation) follow power-law scaling in model size, pre-training compute, resolution, and upcycling, and that scaling test-time denoising steps and ensembling improves accuracy, matching specialized methods with far less data.
Problem
How to scale diffusion models for visual perception tasks (depth, optical flow, amodal segmentation) in terms of training and test-time compute under limited budgets.
Approach
Treats perception tasks as conditional image-to-image diffusion with a DiT backbone, systematically studies power-law scaling of training compute (model size, MoE, resolution, upcycling, pre-training steps) and test-time compute (denoising steps, median ensembling, noise schedule), and trades off reduced pre-training for increased inference compute.
Key results
- Depth: the DiT-XL model with 200 denoising steps and 5-sample median ensembling matches Marigold on Hypersim (AbsRel 13.6 vs 13.5) and beats it on ETH3D (AbsRel 4.8 vs 6.5, δ1 97.8 vs 96.0) using roughly three orders of magnitude less pre-training data and compute.
- Optical flow: on FlyingChairs the model reaches EPE 3.08 with ensembling, comparable to specialized methods such as FlowNetC (2.19) and DeepFlow (3.53), trained on far less data.
- Amodal segmentation: mIoU of 82.9 on COCO-A, 88.6 on Pix2Gestalt and 63.9 on MP3D, matching or exceeding pix2gestalt on MP3D (61.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed diffusion model instead of Marigold | DiT-XL depth estimator · ETH3D test | AbsRel: 6.5 -> 4.8 | Helped |
| Use the proposed diffusion model instead of Marigold | DiT-XL depth estimator · NYUv2, ScanNet, and DIODE | AbsRel: 5.5 -> 6.8; 6.4 -> 7.7; 30.8 -> 31.0 | Hurt |
| Use the proposed diffusion model instead of Marigold | DiT-XL depth estimator · Hypersim | AbsRel: 13.5 -> 13.6 | Hurt |
| Add test-time ensembling (ablation) | DiT-XL optical-flow estimator · FlyingChairs validation | EPE: 3.45 -> 3.08 | Helped |
| Use the proposed diffusion model instead of FlowNetC | DiT-XL optical-flow estimator · FlyingChairs validation | EPE: 2.19 -> 3.45 | Hurt |
| Use the proposed diffusion model instead of pix2gestalt | DiT-XL amodal segmenter · COCO-A and P2G | mIOU: 82.9 -> 82.9; 88.7 -> 88.6 | Mixed |
| Use the proposed diffusion model instead of pix2gestalt | DiT-XL amodal segmenter · MP3D | mIOU: 61.5 -> 63.9 | Helped |
| Increase DiT model size (ablation) | Depth estimator fine-tuned on Hypersim · Hypersim | AbsRel | Helped |
| Increase DiT pre-training steps (ablation) | Depth estimator fine-tuned on Hypersim · Hypersim | AbsRel | Helped |
| Increase image resolution during fine-tuning from 256×256 to 512×512 (ablation) | Depth estimator · Hypersim | AbsRel | Helped |
| Upcycle fine-tuned dense DiT models to sparse MoE models (ablation) | Depth estimator fine-tuned on Hypersim · Hypersim | AbsRel | Helped |
| Increase the number of diffusion sampling steps (ablation) | Depth estimator at inference · Depth estimation | AbsRel | Helped |
Surprising findings
- Increasing test-time compute (denoising steps and ensembling) compensates for pre-training budgets 3-4 orders of magnitude smaller.
- Cosine noise variance schedule outperforms linear and scaled-linear schedules for DDIM on depth under a fixed compute budget.
Limitations
- Optical flow and amodal segmentation are evaluated on limited benchmarks and not all compared against the strongest specialized methods
- Depth ensembling with median compilation requires multiple forward passes, increasing inference cost
- Depth model trained at lower resolution than some baselines
Experimental setup
Datasets: ImageNet-1K Hypersim FlyingChairs pix2gestalt ETH3D NYUv2 ScanNet DIODE COCO-A MP3D
Benchmarks: Hypersim ETH3D NYUv2 ScanNet DIODE FlyingChairs COCO-A Pix2Gestalt MP3D
Metrics: AbsRel Delta1 EPE mIoU
Baselines: Marigold DPT LeReS MiDaS Omnidata DeepFlow FlowNetC FlowNetS pix2gestalt PCNet SAM SD-XL Inpainting
Builds on: DiT Marigold pix2gestalt FlowDiffuser Sparse MoE upcycling
Closest papers by meaning
- What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.92)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.90)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.90)
- DEEM: Diffusion models serve as the eyes of large language models for image perception (ICLR 2025 · similarity 0.89)
- Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception (ICLR 2025 · similarity 0.89)
- DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.