Research map / Detection and segmentation / CNN features
Leveraging RGB-D Data with Cross-Modal Context Mining for Glass Surface Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RGB-D GSD, a 3,009-image RGB-D glass surface dataset, and a two-branch RGB-depth network with cross-modal context mining (CCM) and depth-missing aware attention (DAA) modules that outperforms RGB glass detectors.
Problem
Glass surfaces are visually ambiguous and transparent, so RGB-only glass surface detection fails on ambiguous boundaries and glass-like distractors; depth sensors return noisy or missing depth on glass, which is unexploited.
Approach
Use depth maps as a complementary cue: multi-scale cross-modal context mining over RGB and depth features, plus an attention module that explicitly uses the spatial locations of missing depth (depth-missing map) to localize glass.
Key results
- On RGB-D GSD the proposed model reaches IoU 0.742, F-beta 0.853, MAE 0.043 and BER 9.33, beating the best baseline GSDNet (IoU 0.714, BER 9.73).
- On GSD (RGB-only setting) the model reaches IoU 0.849 vs 0.836 for GSDNet and RFENet.
- Replacing the depth map with a grayscale image or an empty map at inference drops IoU to about 0.33-0.35, showing reliance on real depth.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed RGB-D method rather than the strongest baseline | RGB-D GSD glass surface detection · RGB-D GSD | IoU: 0.714 -> 0.742 | Helped |
| Use the proposed RGB-based method rather than the strongest baseline | GDD glass surface detection · GDD | Fβ: 0.932 -> 0.933 | Helped |
| Use the proposed RGB-based method rather than the strongest baseline | GSD glass surface detection · GSD | IoU: 0.836 -> 0.849 | Helped |
| Add CCM to the base model (ablation) | RGB-D GSD framework · RGB-D GSD | IoU: 0.703 -> 0.727 | Helped |
| Add DAA to the base model (ablation) | RGB-D GSD framework · RGB-D GSD | IoU: 0.703 -> 0.706 | Helped |
| Remove depth-related modules and use RGB only (ablation) | Ours · RGB-D GSD | IoU: 0.742 -> 0.686 | Hurt |
| Replace the depth map with grayscale input at inference (ablation) | Ours · RGB-D GSD | IoU: 0.742 -> 0.325 | Hurt |
| Replace the depth map with a black image at inference (ablation) | Ours · RGB-D GSD | IoU: 0.742 -> 0.352 | Hurt |
| Remove the depth-missing map input from DAA (ablation) | DAA · RGB-D GSD | IoU: 0.742 -> 0.733 | Hurt |
| Apply DAA only to RGB features instead of all three feature types (ablation) | DAA · RGB-D GSD | IoU: 0.742 -> 0.738 | Hurt |
| Apply DAA only to depth features instead of all three feature types (ablation) | DAA · RGB-D GSD | IoU: 0.742 -> 0.729 | Hurt |
| Apply DAA only to cross-modal features instead of all three feature types (ablation) | DAA · RGB-D GSD | IoU: 0.742 -> 0.739 | Hurt |
Surprising findings
- Replacing the depth input with a grayscale or black image at inference collapses performance (IoU ~0.33-0.35), far below the RGB-only variant (0.686).
Limitations
- Fails under low-light scenes.
- Fails with perfect glass transmission (no depth returned) or mirror reflections.
- Gains on GDD are relatively small because existing methods already perform well there.
Experimental setup
Datasets: RGB-D GSD GDD GSD SUN RGB-D 2D-3D-Semantics Matterport3D
Benchmarks: RGB-D GSD GDD GSD
Metrics: IoU F-beta MAE BER
Baselines: SPNet CLNet GDNet EBLNet GSDNet RFENet
Builds on: EBLNet GSDNet ResNeXt-101
Closest papers by meaning
- Exploiting Semantic Relations for Glass Surface Detection (NeurIPS 2022 · similarity 0.91)
- Rich Context Aggregation With Reflection Prior for Glass Surface Detection (CVPR 2021 · similarity 0.91)
- Multi-View Dynamic Reflection Prior for Video Glass Surface Detection (AAAI 2024 · similarity 0.87)
- Cross-Modal Distillation for 2D/3D Multi-Object Discovery from 2D Motion (CVPR 2025 · similarity 0.83)
- Beyond Virtual Points: Depth-Enhanced LiDAR-only 3D Object Detection with Semi-Supervised Learning (Student Abstract) (AAAI 2025 · similarity 0.82)
- Pre-training LiDAR-based 3D Object Detectors through Colorization (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.