Research map / Detection and segmentation / 3D detection and driving
SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SP3D, a two-stage strategy that transfers boundary-shrunk LMM (FastSAM + SemanticSAM) semantic masks onto LiDAR points as seeds, generates dynamic-radius DBSCAN pseudo-boxes, filters them with a DS score, pretrains a detector with them, then fine-tunes on sparse labels.
Problem
Sparsely-supervised 3D object detection degrades sharply when accurate labels are extremely scarce, because the initial detector lacks discriminative features learned from so few annotations.
Approach
Use LMM-generated 2D foreground masks, shrunk to remove ambiguous edges and projected onto point clouds as seed points, to drive dynamic-radius clustering pseudo-labels and a distribution-shape score that replaces confidence in NMS, giving the 3D detector label-free initialization before sparse fine-tuning.
Key results
- On KITTI with 2% labels, CoIn++ with SP3D raises Pedestrian moderate AP from 36.1 to 58.7 and Cyclist moderate AP from 58.4 to 73.1 (Table 1).
- On WOD with 1% annotations, CoIn+SP3D improves Vehicle LEVEL 1 mAP from 39.6 to 46.7 (+7.1).
- In the zero-shot KITTI setting, SP3D reaches 69.71 Car-3D AP@0.7 on Easy versus 60.01 for WS3D-PR (Table 5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SP3D initialization to CoIn++ | CoIn++ with VoxelRCNN · KITTI val split, Car | 3D AP: 70.2 -> 80.5 | Helped |
| Add SP3D to CoIn | CoIn with CenterPoint · WOD validation, Vehicle Level 1 | mAP: +7.1 | Helped |
| Use SP3D for zero-shot detection | 3D object detection on KITTI · KITTI val split, Car-3D | AP: 40.32/9.09 -> 93.75/69.71 | Helped |
| Add DCPG to mask shrink (ablation) | SP3D with CenterPoint · KITTI val split, Car-3D AP@IoU 0.7 | AP: 35.10 -> 38.26; 23.75 -> 29.25; 19.52 -> 24.74 | Helped |
| Add DS score to mask shrink (ablation) | SP3D with CenterPoint · KITTI val split, Car-3D AP@IoU 0.7 | AP: 35.10 -> 40.56; 23.75 -> 28.15; 19.52 -> 22.40 | Helped |
| Add mask shrink to DCPG and DS score (ablation) | SP3D with CenterPoint · KITTI val split, Car-3D AP@IoU 0.7 | AP: 47.23 -> 52.56; 33.40 -> 38.00; 27.13 -> 31.52 | Helped |
| Use distribution constraint instead of meta-shape constraint (ablation) | DS score in SP3D · KITTI val split, Car-3D AP@IoU 0.7 | AP Easy: 44.91 -> 48.48 | Helped |
Surprising findings
- The paper reports a slight drop in Easy-level car precision when SP3D initializes the detector, which the authors attribute to differences between geometry-based pseudo-box conventions and manual annotation conventions.
- Performance degrades significantly when fine-tuning uses annotation rates below 0.1%.
Limitations
- Significant degradation when fine-tuning with annotation rates below 0.1%, possibly due to noise from extremely few labels.
- Slight precision decrease on KITTI Easy car category because pseudo-box conventions differ from manual annotations.
- Pseudo-label quality relies on the accuracy of 2D LMM masks and 2D-3D calibration.
Experimental setup
Datasets: KITTI Waymo Open Dataset
Benchmarks: KITTI val split Waymo Open Dataset validation set
Metrics: 3D AP R40 BEV AP R40 mAP mAPH
Baselines: CoIn CoIn++ SS3D HSSDA HINTED VS3D WS3D SAM3D CM3D CPD CenterPoint VoxelRCNN
Builds on: CoIn CPD FastSAM SemanticSAM DBSCAN
Closest papers by meaning
- Prompt3D: Random Prompt Assisted Weakly-Supervised 3D Object Detection (CVPR 2024 · similarity 0.89)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.89)
- Seg2Box: 3D Object Detection by Point-Wise Semantics Supervision (AAAI 2025 · similarity 0.88)
- LabelAny3D: Label Any Object 3D in the Wild (NeurIPS 2025 · similarity 0.87)
- Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection (AAAI 2024 · similarity 0.87)
- Dual-Perspective Knowledge Enrichment for Semi-supervised 3D Object Detection (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.