Research map / Image generation / Autoregressive and latent generation
Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pixel-Perfect Depth performs monocular depth estimation via flow-matching diffusion transformers directly in pixel space (no VAE), using semantics-prompted DiT and a cascade patch-size design to avoid flying pixels at edges.
Problem
Generative depth estimators that fine-tune Stable Diffusion compress depth maps with a VAE, which introduces flying pixels at object edges and fine details when converted to point clouds; naive pixel-space diffusion is hard to optimize.
Approach
Train a diffusion transformer from scratch directly in pixel space, inject L2-normalized semantic features from a frozen vision foundation model encoder into the DiT tokens (SP-DiT), and use a coarse-to-fine cascade that starts with large patches and halves patch size in later blocks (Cas-DiT).
Key results
- Ours (1024) reaches AbsRel 4.1 on NYUv2 and 6.8 on DIODE, the best among listed generative models on the five zero-shot benchmarks.
- SP-DiT reduces NYUv2 AbsRel from 22.5 (vanilla DiT) to 4.8, and adding Cas-DiT lowers inference time from 0.20s to 0.14s on RTX 4090.
- Edge-aware Chamfer distance on Hypersim is 0.08 for the proposed model versus 0.14 for DepthPro and 0.12 for VAE-reconstructed GT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SP-DiT semantic prompting to vanilla DiT (ablation) | Pixel-Perfect Depth · NYUv2 | AbsRel: 22.5 -> 4.8 | Helped |
| add Cas-DiT to SP-DiT (ablation) | Pixel-Perfect Depth · NYUv2 | AbsRel: 4.8 -> 4.3 | Helped |
| add Cas-DiT to SP-DiT (ablation) | Pixel-Perfect Depth · inference | time (s): 0.20 -> 0.14 | Helped |
| use a pretrained vision encoder for SP-DiT rather than no semantic prompting (ablation) | Pixel-Perfect Depth · NYUv2 | AbsRel: 22.5 -> 6.4 | Helped |
| use VGGT rather than MAE as the SP-DiT encoder (ablation) | Pixel-Perfect Depth · NYUv2 | AbsRel: 6.4 -> 4.7 | Helped |
| use a smaller DiT-Small-based PPD variant (ablation) | Pixel-Perfect Depth · Runtime | Time (ms): 140 -> 40 | Helped |
| use a smaller DiT-Small-based PPD variant (ablation) | Pixel-Perfect Depth · NYUv2 | AbsRel: 4.3 -> 4.5 | Hurt |
| use pixel-space diffusion rather than VAE-reconstructed depth | Pixel-Perfect Depth · Hypersim test | Chamfer Dist.: 0.12 -> 0.08 | Helped |
Surprising findings
- Reconstructing ground-truth depth through a VAE alone (GT(VAE)) yields a larger Chamfer distance than the proposed model, indicating VAE compression itself causes flying pixels.
- REPA-style representation alignment gives only modest gains over vanilla DiT (NYUv2 AbsRel 17.6 vs 22.5) compared with the proposed semantic prompting (4.3).
Limitations
- Lacks temporal consistency on video, causing flickering depth across frames.
- Multi-step diffusion makes inference slower than discriminative models such as Depth Anything v2.
Experimental setup
Datasets: Hypersim UrbanSyn UnrealStereo4K VKITTI TartanAir NYUv2 KITTI ETH3D ScanNet DIODE
Benchmarks: NYUv2 KITTI ETH3D ScanNet DIODE Hypersim
Metrics: AbsRel delta_1 Chamfer Distance (edge-aware point cloud) inference time
Baselines: Marigold GeoWizard DepthFM GenPercept Lotus Depth Anything v2 DepthPro MoGe MiDaS Omnidata DPT REPA
Builds on: DiT Flow Matching REPA Marigold
Closest papers by meaning
- Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis (ICLR 2024 · similarity 0.88)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.88)
- More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models (NeurIPS 2025 · similarity 0.88)
- SharpDepth: Sharpening Metric Depth Predictions Using Diffusion Distillation (CVPR 2025 · similarity 0.87)
- Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (CVPR 2024 · similarity 0.87)
- DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.