Research map / Detection and segmentation / CNN features
DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pretrains an RGB-D backbone on ImageNet-1K image-depth pairs (depth from AdaBins) with RGB-D blocks, then fine-tunes with a lightweight Hamburger decoder for RGB-D semantic segmentation and salient object detection.
Problem
RGB-D segmentation and salient object detection methods rely on RGB-pretrained backbones to encode depth, causing representation mismatch, dense cross-modal interaction during finetuning, and high computational cost.
Approach
Pretrain the backbone itself on image-depth pairs (depth estimated by AdaBins) so RGB and depth features interact inside each building block (GAA, LEA, base module) from pretraining onward, use a smaller channel budget for depth, and decode only RGB features.
Key results
- DFormer-L reaches 57.2% mIoU on NYUDepthv2 with 39.0M params and 65.7G FLOPs, versus 56.9% for CMNeXt (MiT-B4) at 131.9G FLOPs.
- DFormer-B (55.6% mIoU, 29.5M params, 41.9G FLOPs) outperforms CMX (MiT-B2, 54.4%, 66.6M, 67.6G).
- DFormer-T reaches 51.8% mIoU on NYUDepthv2 with only 6.0M params and 11.8G FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain with RGB-depth pairs instead of replacing depth maps with RGB images (ablation) | DFormer-B · NYU DepthV2 | mIoU: 53.3 -> 55.6 | Helped |
| Add RGB-depth pretraining to DFormer-L (ablation) | DFormer-L · NYU DepthV2 | mIoU: 55.4 -> 57.2 | Helped |
| Add depth features to the decoder input (ablation) | DFormer-B · NYU DepthV2 | mIoU: 55.6 -> 55.5 | No clear effect |
| Add the Base module (ablation) | DFormer-S · NYU DepthV2 | mIoU: 52.1 -> 53.6 | Helped |
| Add GAA (ablation) | DFormer-S · NYU DepthV2 | mIoU: 52.3 -> 53.6 | Helped |
| Add LEA (ablation) | DFormer-S · NYU DepthV2 | mIoU: 52.6 -> 53.6 | Helped |
| Increase GAA Q pooling from 3×3 to 7×7 (ablation) | DFormer-S · NYU DepthV2 | mIoU: 52.7 -> 53.6 | Helped |
| Increase GAA Q pooling from 7×7 to 9×9 (ablation) | DFormer-S · NYU DepthV2 | mIoU: 53.6 -> 53.6 | No clear effect |
| Fuse RGB-depth features into K and V in addition to Q (ablation) | DFormer-S GAA · NYU DepthV2 | mIoU: 53.6 -> 53.6 | No clear effect |
| Fuse depth and RGB features into Q (ablation) | DFormer-S GAA · NYU DepthV2 | mIoU: 53.2 -> 53.6 | Helped |
| Use Hadamard fusion instead of addition in LEA (ablation) | DFormer-S · NYU DepthV2 | mIoU: 53.1 -> 53.6 | Helped |
| Use Hadamard fusion instead of concatenation in LEA (ablation) | DFormer-S · NYU DepthV2 | mIoU: 53.3 -> 53.6 | Helped |
Surprising findings
- Using only RGB decoder features (no depth features into the decoder) matches or slightly exceeds using both, 55.6 vs 55.5 mIoU for DFormer-B.
- Depth-only backbone pretrained on depth images outperforms RGB-pretrained backbone by about 15 mIoU when processing depth maps alone.
Limitations
- Depth maps for ImageNet are estimated by a depth model rather than measured, and gains from RGB-D pretraining are smaller for RGB-T and RGB-L modalities.
- Lightweight variants still heavy for mobile deployment according to authors' own future-work discussion.
- Pretraining ablations were run for shortened 100-epoch schedules in some experiments.
Experimental setup
Datasets: ImageNet-1K NYUDepthv2 SUN-RGBD NJU2K NLPR MFNet KITTI-360
Benchmarks: NYUDepthv2 SUN-RGBD DES NLPR NJU2K STERE SIP MFNet KITTI-360
Metrics: mIoU Structure-measure mean absolute error max F-measure max E-measure Params FLOPs latency
Baselines: CMX CMNeXt TokenFusion MultiMAE SPNet HiDANet SA-Gate ACNet
Builds on: ConvNeXt SegNeXt Hamburger AdaBins CMX
Closest papers by meaning
- DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation (CVPR 2025 · similarity 0.92)
- 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans (CVPR 2019 · similarity 0.89)
- RGB-D Salient Object Detection via 3D Convolutional Neural Networks (AAAI 2021 · similarity 0.89)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.86)
- Multi-Modal Contrastive Masked Autoencoders: A Two-Stage Progressive Pre-training Approach for RGBD Datasets (CVPR 2025 · similarity 0.86)
- TGBFormer: Transformer-GraphFormer Blender Network for Video Object Detection (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.