Research map / Point clouds / Point cloud classification
IPDN: Image-enhanced Prompt Decoding Network for 3D Referring Expression Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IPDN, which injects CLIP multi-view image features into 3D point cloud features and uses text-driven prompts to guide query decoding for 3D referring expression segmentation.
Problem
3D referring expression segmentation suffers from feature ambiguity (information loss in point clouds, weak text alignment) and intent ambiguity (all queries treated equally during decoding).
Approach
Fuse CLIP multi-view 2D features into 3D superpoint features via 2D-3D projection and spherical querying (MSE), and generate task-driven prompts from text-query cross-attention scores filtered by prior-layer probabilities to emphasize relevant queries (PAD).
Key results
- On ScanRefer 3D-RES, IPDN reaches 50.2 overall mIoU, 1.9 points above MDIN (48.3).
- On Multi3DRefer 3D-GRES, IPDN reaches 51.7 mIoU, 4.2 points above MDIN (47.5) per the table.
- Ablation: adding MSE lifts mIoU from 48.1 to 49.2 and PAD adds a further 1.0 to 50.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MSE to the model without MSE or PAD (ablation) | IPDN · ScanRefer | mIoU: 48.1 -> 49.2 | Helped |
| Add PAD to the model with MSE (ablation) | IPDN · ScanRefer | mIoU: 49.2 -> 50.2 | Helped |
| Add MSE and PAD (ablation) | IPDN · ScanRefer | mIoU: 48.1 -> 50.2 | Helped |
| Set spatial-attention k to 2 instead of 8 (ablation) | IPDN · ScanRefer | mIoU: 50.2 -> 49.3 | Hurt |
| Set spatial-attention k to 4 instead of 8 (ablation) | IPDN · ScanRefer | mIoU: 50.2 -> 49.4 | Hurt |
| Set spatial-attention k to 6 instead of 8 (ablation) | IPDN · ScanRefer | mIoU: 50.2 -> 49.7 | Hurt |
| Set spatial-attention k to 10 instead of 8 (ablation) | IPDN · ScanRefer | mIoU: 50.2 -> 50.0 | Hurt |
| Set PAD filtering threshold r to 0 instead of 0.75 (ablation) | IPDN · ScanRefer | mIoU: 50.2 -> 49.4 | Hurt |
| Set PAD filtering threshold r to 0.25 instead of 0.75 (ablation) | IPDN · ScanRefer | mIoU: 50.2 -> 49.6 | Hurt |
| Set PAD filtering threshold r to 0.5 instead of 0.75 (ablation) | IPDN · ScanRefer | mIoU: 50.2 -> 49.6 | Hurt |
| Set PAD filtering threshold r to 0.9 instead of 0.75 (ablation) | IPDN · ScanRefer | mIoU: 50.2 -> 50.0 | Hurt |
Surprising findings
- IPDN underperforms in the zero-target scenario with distractors on Multi3DRefer, where it is below MDIN.
Limitations
- Performance drops in the zero-target scenario with distractor objects on Multi3DRefer.
- Relies on multi-view images and camera parameters in addition to point clouds.
Experimental setup
Datasets: ScanRefer Multi3DRefer ScanNet
Benchmarks: ScanRefer Multi3DRefer
Metrics: mIoU Acc@0.25 Acc@0.5
Baselines: ReLA M3DRef-CLIP 3D-STMN MDIN TGNN InstanceRefer 3DRefTR X-RefSeg3D Reanson3D SegPoint MCLN RefMask3D
Builds on: MDIN 3D-STMN EDA RoBERTa CLIP
Closest papers by meaning
- LESS: Label-Efficient and Single-Stage Referring 3D Segmentation (NeurIPS 2024 · similarity 0.86)
- RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation (NeurIPS 2024 · similarity 0.86)
- X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks (AAAI 2024 · similarity 0.85)
- Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.85)
- 3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation (AAAI 2024 · similarity 0.85)
- GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.