Research map / 3D vision / Depth and stereo
DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DFormerv2, an RGB-D encoder whose geometry self-attention uses depth-derived distance priors (depth plus Manhattan spatial distance) as decay weights on attention instead of encoding depth with a separate network, reaching SOTA RGB-D semantic segmentation at lower FLOPs.
Problem
RGB-D semantic segmentation typically encodes depth with separate or unified neural encoders and fuses it, which is computationally costly and does not explicitly exploit the geometric nature of depth.
Approach
Use depth maps directly as geometry priors: compute pairwise depth distances and spatial Manhattan distances between patch tokens, fuse them with learnable memory weights, and apply the resulting decay (beta^G) multiplicatively to self-attention weights, with axial decomposition to reduce cost.
Key results
- DFormerv2-B reaches 57.7% mIoU on NYU DepthV2 with 53.9M params and 67.2G FLOPs, matching GeminiFusion (MiT-B5) at under half the compute.
- DFormerv2-L achieves 58.4% mIoU on NYU DepthV2 with 95.5M params and 124.1G FLOPs, surpassing GeminiFusion by 0.7%.
- On DeLiVER, DFormerv2-L reaches 67.1 mIoU versus 66.9 for GeminiFusion (MiT-B5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add depth prior to self-attention (ablation) | Vanilla attention · NYU DepthV2 | mIoU: 51.7 -> 54.3 | Helped |
| add spatial prior to self-attention (ablation) | Vanilla attention · NYU DepthV2 | mIoU: 51.7 -> 53.5 | Helped |
| combine depth and spatial priors into a geometry prior (ablation) | DFormerv2-S attention · NYU DepthV2 | mIoU: 51.7 -> 56.2 | Helped |
| combine depth and spatial priors into a geometry prior (ablation) | DFormerv2-S attention · SUNRGBD | mIoU: 47.8 -> 51.7 | Helped |
| decompose attention along horizontal and vertical axes (ablation) | Geometry self-attention · NYU DepthV2 | mIoU: 56.2 -> 56.0 | Hurt |
| replace memory-weight fusion with addition (ablation) | Geometry prior generation · NYU DepthV2 | mIoU: 56.2 -> 54.6 | Hurt |
| replace memory-weight fusion with Hadamard product (ablation) | Geometry prior generation · NYU DepthV2 | mIoU: 56.2 -> 54.9 | Hurt |
| replace memory-weight fusion with convolution (ablation) | Geometry prior generation · NYU DepthV2 | mIoU: 56.2 -> 55.8 | Hurt |
| use different decay rates across attention heads sampled in [0.75, 1.0) (ablation) | Geometry self-attention · NYU DepthV2 | mIoU: 55.7 -> 56.0 | Helped |
| use RGB and depth rather than RGB alone (ablation) | DFormerv2 · LUSS | weighted F-measure: 0.818 -> 0.868 | Helped |
| use RGB and depth rather than RGB alone (ablation) | DFormerv2 · LUSS | mean absolute error: 0.054 -> 0.048 | Helped |
Surprising findings
- Depth-only prior with no depth encoder network improves over vanilla attention on RGB-D segmentation, and a learned memory-weight fusion of depth and spatial priors beats addition, Hadamard product, and convolution fusion.
Limitations
- Geometry prior depends on the availability of a depth map at inference time.
- Fusion of depth and spatial priors is a simple learned weighting; the paper notes more advanced fusion may help.
- ImageNet depth maps are produced by an estimator rather than sensors, which may introduce noise in pretraining.
Experimental setup
Datasets: ImageNet-1K NYU DepthV2 SUN-RGBD DeLiVER LUSS
Benchmarks: NYU DepthV2 SUN-RGBD DeLiVER
Metrics: mIoU Params FLOPs inference latency top-1 accuracy weighted F-measure MAE
Baselines: CMX GeminiFusion CMNeXt DFormer TokenFusion MultiMAE AsymFormer Omnivore
Builds on: DFormer Swin Transformer MultiMAE CMX SegFormer
Closest papers by meaning
- DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation (ICLR 2024 · similarity 0.92)
- Depth CNNs for RGB-D Scene Recognition: Learning from Scratch Better than Transferring from RGB-CNNs (AAAI 2017 · similarity 0.87)
- SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation (AAAI 2025 · similarity 0.85)
- Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling (CVPR 2024 · similarity 0.85)
- DeCoTR: Enhancing Depth Completion with 2D and 3D Attentions (CVPR 2024 · similarity 0.85)
- SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representations (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.