Research map / Image generation / Text-to-image diffusion
Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Lotus adapts Stable Diffusion V2 into a single-step, x0-prediction diffusion model with a detail preserver (joint reconstruction task) for zero-shot monocular depth and normal estimation, trained on 59K synthetic samples.
Problem
Existing diffusion-based dense predictors inherit the standard image-generation formulation (epsilon-prediction, multi-step denoising), which is suboptimal and slow for zero-shot dense prediction such as depth and normal estimation.
Approach
Replace epsilon-prediction with x0-prediction, collapse the diffusion process to a single step at t=T, and add a task-switched reconstruction objective (detail preserver) so the fine-tuned model keeps fine detail; Lotus-G keeps noise input (generative) and Lotus-D drops it (deterministic).
Key results
- Lotus-G achieves average rank 1.3 on zero-shot affine-invariant depth vs. 2.1 for Marigold and 2.4 for DepthAnything, with 59K training samples.
- Lotus-G obtains the best normal estimation average rank (1.0) across NYUv2, ScanNet, iBims-1, Sintel and OASIS, ahead of StableNormal (2.1) and DSINE (3.1).
- Ablation shows x0-prediction plus single step plus detail preserver reduces NYUv2 AbsRel from 11.551 (direct adaptation) to 5.555 at 39K data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ε-prediction with x₀-prediction (ablation) | Direct Adaptation · NYUv2 depth | AbsRel: 11.551 -> 8.332 | Helped |
| use a single diffusion time-step (ablation) | Lotus depth adaptation · NYUv2 depth | AbsRel: 8.332 -> 5.587 | Helped |
| add the detail preserver reconstruction task (ablation) | Single-step Lotus depth adaptation · NYUv2 depth | AbsRel: 5.587 -> 5.555 | Helped |
| add Virtual KITTI to training data (ablation) | Lotus depth adaptation · NYUv2 depth | AbsRel: 5.555 -> 5.425 | Helped |
| predict in disparity space (ablation) | Lotus-G · NYUv2 depth | AbsRel: 5.425 -> 5.379 | Helped |
| remove noise input (ablation) | Lotus-G · NYUv2 depth | AbsRel: 5.379 -> 5.123 | Helped |
| remove noise input (ablation) | Lotus-G · ETH3D depth | AbsRel: 5.878 -> 6.147 | Hurt |
| vary the single diffusion time-step from t=1000 to another tested value (ablation) | Single-step depth model · NYUv2 depth | AbsRel: 5.587 -> 5.631; 5.727; 5.663; 5.737 | Hurt |
| use x₀-prediction instead of ε-prediction (ablation) | Marigold without AMRN · NYUv2 depth | AbsRel: 13.110 -> 8.058 | Helped |
| reduce the diffusion process to a single step (ablation) | Marigold without AMRN using ε-prediction · NYUv2 depth | AbsRel: 13.110 -> 6.605 | Helped |
| add the detail preserver (ablation) | Marigold without AMRN, single-step x₀-prediction · KITTI depth | AbsRel: 11.166 -> 10.575 | Helped |
| use Lotus-G rather than Diffusion-E2E-FT | Zero-shot normal estimation · Five-dataset evaluation | Avg. Rank: 1.9 -> 1.0 | Helped |
Surprising findings
- Under AMRN-based Marigold training, the advantage of x0-prediction and single-step largely disappears, suggesting AMRN compensates for the variance issue.
- Lotus-D (no noise input) performs comparably to DepthAnything V2 despite using about 59K vs 62.6M training images.
Limitations
- Performance remains slightly behind DepthAnything, which uses large-scale training data.
- Single-step fine-tuning can cause catastrophic forgetting of detail without the detail preserver, especially on KITTI.
- Evaluated only on zero-shot depth and normal estimation in the main paper; other tasks are shown only in supplementary experiments.
Experimental setup
Datasets: Hypersim Virtual KITTI NYUv2 ScanNet KITTI ETH3D DIODE iBims-1 Sintel OASIS
Benchmarks: NYUv2 KITTI ETH3D ScanNet DIODE iBims-1 Sintel OASIS
Metrics: AbsRel delta1 delta2 mean angular error 11.25° accuracy 30° accuracy Avg. Rank
Baselines: Marigold GeoWizard GenPercept Diffusion-E2E-FT StableNormal DepthAnything V2 DepthAnything DSINE Omnidata V2 DPT HDN
Builds on: Stable Diffusion Marigold DDIM v-prediction
Closest papers by meaning
- Exploiting Diffusion Prior for Generalizable Dense Prediction (CVPR 2024 · similarity 0.92)
- What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.90)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.89)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.89)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.