Research map / Point clouds / Segmentation and detection
One for All: Multi-Domain Joint Training for Point Cloud Based 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OneDet3D, a single-parameter-set point cloud 3D detector trained jointly on indoor (SUN RGB-D, ScanNet) and outdoor (KITTI, nuScenes) data, using domain-aware partitioning of normalization and context layers plus CLIP-based language-guided classification.
Problem
Multi-domain joint training for point cloud 3D object detection suffers from severe domain interference (data-level scale/sensor differences and category-level label-space conflicts) between indoor and outdoor datasets, so existing detectors only train and test on a single domain.
Approach
A fully sparse voxel backbone with an anchor-free center-based head, where domain-specific scale/shift parameters in normalization layers and indoor-only global context modules are selected by a domain router, and classification combines frozen CLIP text embeddings with a shared class-agnostic sparse-conv branch.
Key results
- Joint training on SUN RGB-D, ScanNet, KITTI, nuScenes gives 65.0 AP25 on SUN RGB-D (vs FCAF3D 63.8) and 81.0 AP on nuScenes with one parameter set.
- Cross-domain S3DIS AP25 rises from 43.5 (SUN single-dataset) to 53.5 when trained on all four datasets.
- Open-vocabulary novel AP on SUN RGB-D reaches 12.59 vs CoDA's 6.65.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| domain-aware scatter partitioning (per-domain normalization scale/shift) (ablation) | OneDet3D multi-dataset joint training · SUN RGB-D + KITTI joint | AP_25 SUN RGB-D: 61.9 -> 63.2 | Helped |
| context partitioning applied to indoor domains only (ablation) | OneDet3D multi-dataset joint training · SUN RGB-D + KITTI joint, S3DIS unseen | AP_25 SUN RGB-D and S3DIS: SUN 63.2 -> 63.9 (with SP); S3DIS 44.2 -> 45.8 | Helped |
| context partitioning applied to both indoor and outdoor (ablation) | OneDet3D multi-dataset joint training · SUN RGB-D + KITTI joint | AP_m KITTI: 83.2 (no CP) -> 78.6 | Hurt |
| language-guided classification (frozen CLIP embeddings combined with 3D sparse conv classifier) (ablation) | OneDet3D multi-dataset joint training · SUN RGB-D + KITTI joint, S3DIS unseen | AP_25 S3DIS: 45.8 -> 47.8 | Helped |
| classification with frozen CLIP embeddings only (no sparse conv branch) (ablation) | OneDet3D multi-dataset joint training · SUN RGB-D + KITTI joint, S3DIS unseen | AP_25: SUN 63.9 -> 60.2; S3DIS 45.8 -> 25.8 | Hurt |
| trainable CLIP embeddings instead of frozen (ablation) | OneDet3D classification · SUN RGB-D + KITTI joint, S3DIS unseen | AP_25: SUN 64.0 vs 64.4 combined; S3DIS 46.0 vs 47.8 | No clear effect |
| focal loss replaced by soft focal loss with BEV IoU target (ablation) | OneDet3D classification loss · SUN RGB-D + KITTI joint | convergence and AP_25: focal loss not converged -> 64.4 | Helped |
| multi-dataset joint training versus single-dataset training | OneDet3D · SUN RGB-D and KITTI (in-domain) | AP_25 SUN RGB-D / AP_h KITTI: SUN 63.2 -> 65.0; KITTI AP_h 80.0 -> 82.3 | Helped |
| multi-dataset joint training of existing detectors (VoxelNeXt, FSD v2, SAFDNet, Uni3D) | prior 3D detectors · SUN RGB-D, ScanNet, KITTI, nuScenes | AP: e.g., VoxelNeXt SUN 18.1 -> 8.3; SAFDNet KIT 80.3 -> 38.7 | Hurt |
| multi-dataset joint training with language-guided open-vocabulary setup | OneDet3D vs CoDA · SUN RGB-D and ScanNet open-vocabulary | AP_novel: ScanNet 6.02 (CoDA) -> 15.52; SUN 6.65 -> 12.59 | Helped |
| multi-dataset training versus single-dataset training (open-vocabulary) | OneDet3D · SUN RGB-D and ScanNet open-vocabulary | AP_all: SUN 18.45 -> 20.22; ScanNet 18.29 -> 19.77 | Helped |
| adding KIT and nuS joint training data to SUN and ScanNet | OneDet3D cross-domain · S3DIS and Waymo unseen | AP_25 S3DIS and AP^3D Waymo: S3DIS AP_25 52.6 -> 53.5; Waymo AP^3D 40.3 -> 41.1 (with SUN, ScanNet added) | Helped |
Surprising findings
- Applying context partitioning to outdoor data hurts KITTI AP (AP_m drops 3.8%), so it is used for indoor only.
- Focal loss fails to converge once frozen CLIP embeddings are used for classification, requiring soft focal loss with BEV IoU targets.
Limitations
- Multi-dataset joint training degrades performance on some datasets relative to single-dataset training in the baseline setting, and naive joint training causes interference.
- Frozen CLIP embeddings are not usable for classification without the shared class-agnostic branch and require a special loss.
- Outdoor evaluation focuses only on the car category.
Experimental setup
Datasets: SUN RGB-D ScanNet KITTI nuScenes S3DIS Waymo
Benchmarks: SUN RGB-D ScanNet KITTI nuScenes S3DIS Waymo
Metrics: AP25 AP50 AP easy/moderate/hard AP nuScenes AP3D APBEV AP novel/base/all
Baselines: VoteNet H3DNet GroupFree FCAF3D SECOND PointPillar PointRCNN Part-A2 PV-RCNN CenterPoint VoxelNeXt UVTR
Builds on: FCAF3D CLIP mmdetection3D soft focal loss CoDA
Closest papers by meaning
- UniDet3D: Multi-dataset Indoor 3D Object Detection (AAAI 2025 · similarity 0.88)
- Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection (NeurIPS 2024 · similarity 0.88)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.87)
- iDet3D: Towards Efficient Interactive Object Detection for LiDAR Point Clouds (AAAI 2024 · similarity 0.87)
- Towards Learning Group-Equivariant Features for Domain Adaptive 3D Detection (NeurIPS 2024 · similarity 0.86)
- OneFormer3D: One Transformer for Unified Point Cloud Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.