Research map / Image generation / Text-to-image diffusion
More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Starting from fixed PixArt-XL and FLUX.1-dev text-to-image DiTs, the authors insert pluggable converter blocks shared within groups of transformer layers (Group Reuse Mechanism) so the model does zero-shot depth (and normal) estimation while keeping image generation intact.
Problem
Generative depth estimation via full-parameter fine-tuning of text-to-image diffusion models destroys their image generation ability, while existing unified generation+depth models are resource-intensive.
Approach
Keep the pretrained T2I DiT frozen and insert learnable converter blocks (copies of the T2I block, initialized from the group's first block, with cross-attention removed and FFN expansion reduced) before groups of T2I blocks; a converter is shared within each group, and skipping converters restores the T2I model.
Key results
- MERGE-L (FLUX-based, 1.4B params, ~12% trainable) achieves 5.9 A.Rel and 95.4% δ1 on NYUv2, beating OneDiffusion (6.8 / 95.2) which uses 100M training samples.
- MERGE-B (PixArt-based, 110M additional params) reduces LoRA's NYUv2 A.Rel from 8.7 to 7.5 at matched parameter count.
- Group reuse with 14 groups reduces params by about half versus per-block converters (110M vs 224M) with A.Rel 7.5 vs 7.0; removing the GRE with 14 groups drops A.Rel to 15.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove cross-attention from the converter (ablation) | MERGE converter (PixArt, no GRE) · NYUv2 | A.Rel / δ1: A.Rel 6.9 vs 6.9; δ1 94.6 vs 94.8; ~25% fewer params | No clear effect |
| Reduce converter FFN expansion from 4 to 1 (ablation) | MERGE converter · NYUv2 | A.Rel / δ1: A.Rel 7.0 vs 6.9 (setting A) ; δ1 94.7 vs 94.8; ~36% fewer converter params | No clear effect |
| Remove self-attention and FFN from converter (keep only one component) (ablation) | MERGE converter · NYUv2 | A.Rel / δ1: Setting C A.Rel 7.6, δ1 93.6; Setting D A.Rel 7.4, δ1 94.2 vs setting E 7.0 / 94.7 | Hurt |
| Group Reuse Mechanism (sharing one converter per 2-block group) vs one converter per group without sharing (ablation) | MERGE-B · NYUv2 | A.Rel / δ1: A.Rel 15.6 -> 7.5; δ1 78.8 -> 94.2 at same 110M params | Helped |
| Group Reuse Mechanism: halve converter count (14 groups, 110M) vs 28 unshared converters (224M) (ablation) | MERGE-B · NYUv2 | A.Rel / δ1: 7.5/94.2 vs 7.0/94.7 at about half the params | Mixed |
| Increase number of groups (fewer blocks shared per converter) with GRE (ablation) | MERGE-B · NYUv2 | A.Rel / δ1: 4 groups 9.3/91.0; 7 groups 7.8/93.5; 14 groups 7.5/94.2 | Helped |
| Stack multiple converters per block (ablation) | MERGE-B · NYUv2 | A.Rel / δ1: 1 converter 7.5/94.2; 2 converters 7.6/93.8; 3 converters 8.2/92.9 | Hurt |
| Initialize converter from pre-trained T2I block instead of random (ablation) | MERGE-B · NYUv2 | A.Rel / δ1: A.Rel 7.9 -> 7.5; δ1 93.5 -> 94.2 | Helped |
| Replace empty prompt with LLaVA dense captions (ablation) | MERGE-B · NYUv2 | A.Rel / δ1: A.Rel 7.5 -> 7.3; δ1 94.2 -> 94.4 (small gain, costly captions) | Helped |
| Replace empty prompt with fixed 'Depth map' prompt (ablation) | MERGE-B · NYUv2 | A.Rel / δ1: A.Rel 7.4 vs 7.5; δ1 94.3 vs 94.2 | No clear effect |
| Parameter-efficient converter (MERGE-B) vs LoRA/DoRA rank 128 at matched 110M params | PixArt-XL-2-512x512 fine-tuning · NYUv2, ScanNet, DIODE | A.Rel / δ1: NYUv2 A.Rel 7.5 vs 8.6-8.7; ScanNet 9.9 vs 10.6-10.8; DIODE 32.5 vs 32.8-32.9 | Helped |
| Play-and-plug converters (frozen T2I, ~18% params) vs full-parameter fine-tuning (Marigold-P) | PixArt-XL-2-512x512 · NYUv2, ScanNet, DIODE | A.Rel / δ1: NYUv2 7.5/94.2 vs 7.4/94.2; ScanNet 9.9/89.8 vs 9.5/90.1; DIODE 32.5/74.9 vs 31.5/76.0 | Mixed |
Surprising findings
- Cross-attention in the converter is redundant when the text prompt is empty; removing it leaves depth performance essentially unchanged while cutting about 25% of converter parameters.
- Stacking multiple converters before each group degrades performance (A.Rel rises from 7.5 to 8.2 with 3 converters), attributed to optimization difficulty from increased depth.
Limitations
- Semantic segmentation is highly challenging with the approach; one-shot RGB ID mapping is unstable due to denoising randomness.
- Remains behind discriminative DepthAnything v2 on outdoor DIODE benchmark (A.Rel 31.4 for FLUX version vs 6.5 for DA-v2 on DIODE).
- Bit-encoding methods like LDMSeg are incompatible with the existing pretrained VAE.
Experimental setup
Datasets: Hypersim Virtual KITTI NYUv2 ScanNet DIODE iBims-1 Sintel
Benchmarks: NYUv2 ScanNet DIODE iBims-1 Sintel
Metrics: A.Rel delta1 mean angular error 11.25 degree accuracy
Baselines: Marigold GeoWizard DepthFM Lotus JointNet UniCon OneDiffusion DepthAnything v2 LoRA DoRA StableNormal
Builds on: Marigold Stable Diffusion PixArt-alpha FLUX LoRA
Closest papers by meaning
- DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching (AAAI 2025 · similarity 0.88)
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.88)
- Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
- Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (CVPR 2024 · similarity 0.87)
- Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction (ICLR 2025 · similarity 0.87)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.