Research map / Detection and segmentation / Weakly supervised detection and segmentation
Prompt3D: Random Prompt Assisted Weakly-Supervised 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Prompt3D generates synthetic indoor scenes from Shap-E shapes built from ChatGPT-generated prompts, then aligns class prototypes of box proposal features between synthetic and real scenes to train weakly supervised (position-level) 3D object detectors on ScanNet.
Problem
Weakly supervised 3D indoor object detection with only position-level labels (object centers and class tags) suffers from limited object diversity in synthetic scenes built from standard shape datasets like ModelNet40 and a domain gap between virtual and real scenes.
Approach
Use a text-to-3D generator (Shap-E) with many random prompts per category to build a large, diverse shape library for assembling synthetic scenes, and transfer knowledge via prototypical (class-wise) alignment of box proposal features rather than adversarial global feature alignment.
Key results
- With VoteNet, Prompt3D improves over the prior weakly supervised SOTA BR by 5.4% mAP and reduces the gap to fully supervised to 7.7% mAP@0.25.
- With GroupFree3D, Prompt3D reaches 51.2% mAP@0.25 versus 54.2% fully supervised (about 3% gap) and improves by 8.7% mAP over BR per the abstract.
- Ablation on VoteNet: PPFA alignment on top of the prompt-based scene setup reaches 37.4 mAP@0.25 versus 35.5 with adversarial feature alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Prompt3D instead of WSBP | VoteNet · ScanNet val | mAP@0.25: 36.1 -> 37.4 | Helped |
| use Prompt3D instead of BRM | GroupFree3D · ScanNet val | mAP@0.25: 47.1 -> 51.2 | Helped |
| use prompt-based 3D shapes instead of ModelNet40 shapes (ablation) | VoteNet weakly supervised detection with BR assembly · ScanNet val | mAP@0.25: 30.6 -> 31.1 | Helped |
| use prompt-based 3D shapes instead of ModelNet40 shapes (ablation) | VoteNet weakly supervised detection with Prompt3D assembly · ScanNet val | mAP@0.25: 35.0 -> 37.4 | Helped |
| use Prompt3D scene assembly instead of BR assembly (ablation) | VoteNet with ModelNet40 shapes · ScanNet val | mAP@0.25: 30.6 -> 35.0 | Helped |
| use Prompt3D scene assembly instead of BR assembly (ablation) | VoteNet with prompt-based shapes · ScanNet val | mAP@0.25: 31.1 -> 37.4 | Helped |
| add PPFA instead of using AFA (ablation) | VoteNet Prompt3D · ScanNet val | mAP@0.25: 35.5 -> 37.4 | Helped |
| align aggregated voting features instead of seed features (ablation) | VoteNet PPFA · ScanNet val | mAP@0.25: 33.2 -> 37.4 | Helped |
| align aggregated voting features instead of voting features (ablation) | VoteNet PPFA · ScanNet val | mAP@0.25: 35.3 -> 37.4 | Helped |
| align 5 head features instead of 0 head features (ablation) | GroupFree3D PPFA · ScanNet val | mAP@0.25: 44.3 -> 51.2 | Helped |
| align 5 head features instead of 1 head features (ablation) | GroupFree3D PPFA · ScanNet val | mAP@0.25: 47.4 -> 51.2 | Helped |
| align 5 head features instead of 2 head features (ablation) | GroupFree3D PPFA · ScanNet val | mAP@0.25: 48.7 -> 51.2 | Helped |
Surprising findings
- Weakly supervised GroupFree3D with Prompt3D surpasses the fully supervised baseline on several classes (door, laptop, stool, toilet).
- The paper's text reports a 4.1% gain over BR for GroupFree3D while the abstract claims 8.7% mAP, an internal inconsistency.
Limitations
- Relies on a generative model (Shap-E) and ChatGPT-generated prompts; scene realism still differs from real scenes.
- Requires position-level labels (object centers and class tags), so it is not fully label-free.
- Evaluated only on ScanNet with 22 categories.
Experimental setup
Datasets: ScanNet ModelNet40
Benchmarks: ScanNet
Metrics: mAP@0.25 mAP@0.5
Baselines: WSB WSBP BRP BRM FSB Adversarial Feature Alignment
Builds on: VoteNet GroupFree3D Shap-E BR (Xu et al. weakly supervised 3D detection)
Closest papers by meaning
- SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Prompts (CVPR 2025 · similarity 0.89)
- LabelAny3D: Label Any Object 3D in the Wild (NeurIPS 2025 · similarity 0.87)
- Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection (AAAI 2024 · similarity 0.86)
- PromptDet: A Lightweight 3D Object Detection Framework with LiDAR Prompts (AAAI 2025 · similarity 0.85)
- UniDet3D: Multi-dataset Indoor 3D Object Detection (AAAI 2025 · similarity 0.85)
- Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.