Research map / Detection and segmentation / 3D detection and driving
Seg2Box: 3D Object Detection by Point-Wise Semantics Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Seg2Box trains LiDAR 3D object detectors using only semantic point labels, generating box pseudo-labels via multi-frame multi-scale DBSCAN clustering with a meta-shape/fitting score, then refining them with semantic-guided iterative self-training.
Problem
3D object detection from LiDAR without bounding box labels, supervised only by point-wise semantic labels; challenges are incomplete geometry from sparse/occluded points and boundary ambiguity from truncated or adjacent objects, which produce inaccurate box pseudo-labels.
Approach
Replace box annotations with semantic labels: generate box candidates by clustering semantic points at multiple radii across consecutive frames (static areas only), select the best candidate with a new MSF-Score (occupancy, alignment, meta-shape), and then run self-training that filters background-mislabeled foreground points, filters inconsistent predictions by semantics, refines static-object bo
Key results
- On WOD validation, Seg2Box reaches 90.56 vehicle 3D AP L1 at IoU 0.5 (vs 93.75 for fully supervised CenterPoint), and Seg2Box + Init beats the best baseline by 21.8% in L2 mAP.
- On nuScenes validation, Seg2Box achieves 46.3 mAP vs 36.0 for the best baseline DBSCAN + Sem + St (vs 54.6 for fully supervised CenterPoint).
- Pseudo-label 3D recall at IoU 0.5 on WOD is 57.49 for Seg2Box vs 28.30 for OYSTER.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-frame clustering instead of single-frame clustering (ablation) | Seg2Box pseudo-label generation · WOD val, Vehicle 3D AP L2 | 3D AP: 17.46 -> 44.41 | Helped |
| add multi-scale clustering (ablation) | Multi-Frame Clustering · WOD val, Vehicle 3D AP L2 | 3D AP: 44.41 -> 48.71 | Helped |
| add SGIM-ST self-training (ablation) | MFMS-C detector · WOD val, Vehicle 3D AP L2 | 3D AP: 48.71 -> 54.71 | Helped |
| add Box-Absent Foreground Filter (ablation) | SGIM-ST · WOD val, Vehicle 3D AP L2 | 3D AP: 43.66 -> 48.70 | Helped |
| add MSF-Weighted Detection Loss (ablation) | SGIM-ST with BAF · WOD val, Vehicle 3D AP L2 | 3D AP: 48.70 -> 49.71 | Helped |
| add Semantic Consistency Box Filter (ablation) | SGIM-ST with BAF and MSFLoss · WOD val, Vehicle 3D AP L2 | 3D AP: 49.71 -> 51.87 | Helped |
| add Spatial-Temporal Consistency Box Fine-Tuning (ablation) | SGIM-ST with BAF, MSFLoss, and SCF · WOD val, Vehicle 3D AP L2 | 3D AP: 51.87 -> 54.71 | Helped |
| add self-training to Seg2Box Init (ablation) | Seg2Box · WOD val, Vehicle 3D AP L2, IoU 0.5 | 3D AP: 76.46 -> 83.73 | Helped |
| add self-training to Seg2Box Init (ablation) | Seg2Box · nuScenes val | mAP: 44.2 -> 46.3 | Helped |
| use Seg2Box rather than OYSTER + Sem | 3D detector trained with semantic labels · WOD val, Vehicle 3D AP L2, IoU 0.5 | 3D AP: 39.20 -> 83.73 | Helped |
| use Seg2Box rather than DBSCAN + Sem + St | 3D detector trained with semantic labels · nuScenes val | mAP: 36.0 -> 46.3 | Helped |
Limitations
- Moving objects are not refined by the spatio-temporal step because tracking results are poor.
- Pseudo-label quality still degrades for sparse, distant, and truncated objects; performance on pedestrians and cyclists lags fully supervised detection.
- nuScenes evaluation ignores velocity and attribute metrics since they cannot be derived from semantic labels.
Experimental setup
Datasets: Waymo Open Dataset nuScenes
Benchmarks: Waymo Open Dataset nuScenes
Metrics: 3D AP L1 3D AP L2 mAP NDS ATE ASE AOE 3D recall 3D precision
Baselines: DBSCAN + Sem DBSCAN + Sem + St OYSTER + Sem CenterPoint (Boxes Sup)
Builds on: DBSCAN CenterPoint OYSTER OpenPCDet Box2Mask
Closest papers by meaning
- MixSup: Mixed-grained Supervision for Label-efficient LiDAR-based 3D Object Detection (ICLR 2024 · similarity 0.88)
- SeMoLi: What Moves Together Belongs Together (CVPR 2024 · similarity 0.88)
- SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Prompts (CVPR 2025 · similarity 0.88)
- 3D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentation (CVPR 2025 · similarity 0.87)
- Semi-supervised 3D Object Detection with PatchTeacher and PillarMix (AAAI 2024 · similarity 0.87)
- Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.