Research map / Point clouds / Segmentation and detection
Pre-training LiDAR-based 3D Object Detectors through Colorization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Grounded Point Colorization (GPC), a pre-training method that colorizes LiDAR points with ground-truth colors given as hints to a seed subset, using classification with balanced softmax, and evaluates on KITTI and Waymo fine-tuning.
Problem
Pre-training LiDAR-based 3D object detectors with limited labels; naive colorization of LiDAR points from camera colors suffers from inherent color variation and selection bias, slowing learning and degrading transfer.
Approach
Pre-train a point-based backbone by colorizing LiDAR points, but provide ground-truth colors for a seed subset of points as hints to a color decoder, grounding colorization on color variation; colors are quantized via K-means and predicted as classes with balanced softmax.
Key results
- On 20% of KITTI with PointRCNN, GPC reaches 70.7 overall moderate AP versus 69.3 for training from scratch on 100% of the data (scratch 69.7 is the 100% baseline mentioned in text).
- With Waymo pre-training, 5% KITTI overall moderate AP improves from 55.2 to 63.9 (PointRCNN).
- Fine-tuning on 100% KITTI yields a gain of +1.2 overall moderate AP (69.7 to 70.9) over scratch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train with GPC instead of training from scratch | PointRCNN · KITTI val, 5% labeled KITTI | Overall moderate AP: 55.2 to 62.7 | Helped |
| Pre-train with GPC instead of training from scratch | PointRCNN · KITTI val, 100% labeled KITTI | Overall moderate AP: 69.7 to 70.9 (+1.2) | Helped |
| Pre-train with GPC instead of training from scratch | PointRCNN · KITTI val, 5% labeled KITTI; Waymo pre-training | Overall moderate AP: 55.2 to 63.9 | Helped |
| Use 20% of points as color hints rather than no hints (ablation) | GPC · KITTI val, 5% labeled KITTI | Overall moderate AP: 44.1 to 61.9 | Helped |
| Remove color hints (ablation) | GPC · KITTI val, 5% labeled KITTI | Overall moderate AP: 62.7 to 44.1 | Hurt |
| Use modified Balanced Softmax instead of MSE, SmoothL1, or CrossEntropy (ablation) | GPC colorization · KITTI val, 5% labeled KITTI | Overall moderate AP: 60.1, 60.6, and 60.0 to 62.7 | Helped |
| Use more balanced point sampling rather than random sampling (ablation) | GPC colorization · KITTI val, 5% labeled KITTI | Overall moderate AP: 62.7 to 62.5 | No clear effect |
| Increase Waymo pre-training from 1 to 12 epochs (ablation) | GPC · KITTI val, Waymo pre-training | Overall moderate AP: 60.7 to 63.9 | Helped |
| Concatenate RGB color directly with xyz detector input (ablation) | PointRCNN · KITTI val, 5% labeled KITTI | Overall moderate AP: 55.2 to 49.0 | Hurt |
Surprising findings
- Naive RGB regression or concatenating RGB with xyz as input without hints performs worse than training from scratch.
- PointContrast underperforms scratch on 5% KITTI (47.9 vs 55.2 overall moderate AP).
Limitations
- Gains on full-data fine-tuning are modest (about +1.2 AP on KITTI).
- Fair comparison with other image-based baselines is acknowledged as difficult due to differing data processing and architectures.
- Point sampling variants did not yield significant improvement.
Experimental setup
Datasets: KITTI Waymo Open Dataset
Benchmarks: KITTI Waymo Open Dataset
Metrics: AP (easy, moderate, hard) overall AP AP APH
Baselines: scratch PointContrast ProposalContrast DepthContrast PatchContrast PPKT SLidR TriCC STRL GCC-3D ALSO
Builds on: PointRCNN IA-SSD PointNet++ Balanced Softmax PointContrast
Closest papers by meaning
- Multimodal Virtual Point 3D Detection (NeurIPS 2021 · similarity 0.90)
- LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving (CVPR 2019 · similarity 0.89)
- Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data (CVPR 2022 · similarity 0.89)
- Not All Points Are Equal: Learning Highly Efficient Point-Based Detectors for 3D LiDAR Point Clouds (CVPR 2022 · similarity 0.88)
- Beyond Virtual Points: Depth-Enhanced LiDAR-only 3D Object Detection with Semi-Supervised Learning (Student Abstract) (AAAI 2025 · similarity 0.87)
- GBlobs: Explicit Local Structure via Gaussian Blobs for Improved Cross-Domain LiDAR-based 3D Object Detection (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.