Research map / Detection and segmentation / CNN features
PDDM: Pseudo Depth Diffusion Model for RGB-PD Semantic Segmentation Based in Complex Indoor Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RGB-PD segmentation using pseudo depth from monocular depth estimators, aggregated by a plug-and-play PDAM, and a PDDM built on Stable Diffusion features with pseudo depth fused as structured noise.
Problem
RGB-D semantic segmentation requires costly depth sensors and alignment, and raw depth is noisy; integrating pseudo depth from depth estimators into segmentation, especially with diffusion-based backbones, is unexplored.
Approach
Replace real depth with an aggregation of multiple pseudo depth maps (Depth Anything, Meta Prompt-SD, Marigold) via a channel/spatial attention module (PDAM), and fuse the compressed pseudo depth into a frozen-VAE-plus-Stable-Diffusion UNet feature extractor by adding it, in place of Gaussian noise, at diffusion step t=0.
Key results
- PDDM reaches 63.60 mIoU on NYUv2 single-scale and 64.18 with multi-scale, +6.98 mIoU over the best prior method in the table.
- On SUNRGB-D, PDDM achieves 54.44 mIoU single-scale and 55.11 multi-scale, +2.11 mIoU over prior methods.
- Pseudo depth from three estimators aggregated with PDAM (63.60 mIoU) outperforms real depth input to the same diffusion baseline (62.05 RGB-only baseline; real depth variants reported separately).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add pseudo depth using early fusion (ablation) | PDDM · NYUv2 | mIoU: 62.05 -> 63.60 | Helped |
| Use multi-scale inference (ablation) | PDDM · NYUv2 | mIoU: 63.60 -> 64.18 | Helped |
| Use multi-scale inference (ablation) | PDDM · SUNRGB-D | mIoU: 54.44 -> 55.11 | Helped |
| Replace RGB-only input with real depth (ablation) | PDDM · NYUv2 | mIoU: 62.05 -> 63.10 | Helped |
| Add Depth Anything pseudo depth (ablation) | PDDM · NYUv2 | mIoU: 62.05 -> 62.80 | Helped |
| Add Meta Prompt-SD pseudo depth (ablation) | PDDM · NYUv2 | mIoU: 62.05 -> 63.03 | Helped |
| Aggregate three pseudo depth maps with PDAM instead of simple addition (ablation) | PDDM · NYUv2 | mIoU: 62.78 -> 63.60 | Helped |
| Add pseudo depth at RGB:PD 1:0.03 as structured noise (ablation) | PDDM · NYUv2 | mIoU: 62.05 -> 63.60 | Helped |
| Increase the pseudo-depth share from RGB:PD 0.95:0.05 to 0.6:0.4 (ablation) | PDDM · NYUv2 | mIoU: 63.34 -> 61.89 | Hurt |
| Use timestep 50 instead of timestep 0 (ablation) | PDDM · NYUv2 | mIoU: 63.60 -> 62.77 | Hurt |
| Combine timesteps 0, 50, and 100 instead of using timestep 0 (ablation) | PDDM · NYUv2 | mIoU: 63.60 -> 63.08 | Hurt |
| Use mid-term fusion instead of early fusion (ablation) | PDDM · NYUv2 | mIoU: 63.60 -> 62.98 | Hurt |
Surprising findings
- Pseudo depth from Meta Prompt-SD sometimes yields better NYUv2 mIoU than real depth across several RGB-D backbones.
- Simple addition of multiple pseudo depth maps degrades performance relative to PDAM aggregation (e.g., 62.78 vs 63.60 for PDDM).
Limitations
- Relies on pseudo depth from external depth estimators, so quality depends on those models.
- Fusion weight of pseudo depth is tuned manually (RGB:PD ratio) and relies on t=0 setting.
Experimental setup
Datasets: NYUv2 SUNRGB-D
Benchmarks: NYUv2 SUNRGB-D
Metrics: pixel accuracy mean pixel accuracy mIoU
Baselines: EMSANet CMX CMNeXt DFormer ODISE SA-Gate SGNet CEN ShapeConv PGDENet FRNet TokenFusion
Builds on: Stable Diffusion Mask2Former Meta Prompt-SD Depth Anything Marigold
Closest papers by meaning
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.85)
- Geometric Exploitation for Indoor Panoramic Semantic Segmentation (NeurIPS 2024 · similarity 0.84)
- DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation (ICLR 2024 · similarity 0.84)
- DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation (CVPR 2025 · similarity 0.84)
- $\mathbf{D^{3}S^{2}}$: Diffusion-Guided Dataset Distillation for Semantic Segmentation (NeurIPS 2026 · similarity 0.84)
- Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.