Research map / Detection and segmentation / 3D detection and driving
Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DRET, a physics-grounded two-stage rain simulator for LiDAR point clouds, and SRKD, a sunny-to-rainy knowledge distillation framework with adaptive instance weighting, response distillation and noise-aware correction, improving 3D detectors in rain on Waymo.
Problem
LiDAR 3D object detectors degrade badly in rain due to dense rain noise and missing points, and existing rain simulations (LISA, SPRAY) are unrealistic while rainy training data is scarce (about 0.6% of WOD perception samples).
Approach
Combine a dynamics-based splash simulation (Unity3D particles with Perlin-noise wind) with rainy-environment LiDAR physics to produce realistic rain point clouds, then distill from a sunny teacher detector to a rainy student using similarity-weighted instance distillation, precise response distillation, and a noise-aware prediction correction head.
Key results
- On WOD-DA rainy data, DRET-Aug plus SRKD improves All(L2-mAP) by 4.3 for Voxel-RCNN, 3.0 for PV-RCNN++, and 3.2 for DSVT.
- DRET-Aug alone improves All(L2-mAP) by 0.8, 1.0 and 0.6 for Voxel-RCNN, PV-RCNN++ and DSVT.
- SRKD also yields small sunny-weather gains on WOD-P validation, e.g., Voxel-RCNN Vehicle L2 mAP rising from 69.8 to 70.0 and DSVT All L2 mAP from 73.4 to 73.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DRET-Aug and SRKD | Voxel-RCNN · WOD-DA rainy, All(L2) | mAP: 55.8 -> 60.1 | Helped |
| Add DRET-Aug and SRKD | PV-RCNN++ · WOD-DA rainy, All(L2) | mAP: 56.8 -> 59.8 | Helped |
| Add DRET-Aug and SRKD | DSVT · WOD-DA rainy, All(L2) | mAP: 60.4 -> 63.6 | Helped |
| Add DRET-Aug and SRKD | 3D detectors · WOD-P sunny, All(L2) | mAP: 70.6 -> 71.0; 70.8 -> 71.4; 73.4 -> 73.7 | Mixed |
| Use DRET-Aug instead of no rain augmentation (ablation) | Voxel-RCNN · WOD-DA rainy, All(L2) | mAP: 55.8 -> 56.6 | Helped |
| Use DRET-Aug instead of no rain augmentation (ablation) | PV-RCNN++ · WOD-DA rainy, All(L2) | mAP: 56.8 -> 57.8 | Helped |
| Use DRET-Aug instead of no rain augmentation (ablation) | DSVT · WOD-DA rainy, All(L2) | mAP: 60.4 -> 61.0 | Helped |
| Add AWID (ablation) | Voxel-RCNN + DRET-Aug · WOD-DA rainy, All(L1) | mAP: 63.3 -> 65.3 | Helped |
| Add PRD after AWID (ablation) | Voxel-RCNN + DRET-Aug · WOD-DA rainy, All(L1) | mAP: 65.3 -> 66.2 | Helped |
| Add NAPC after AWID and PRD (ablation) | Voxel-RCNN + DRET-Aug · WOD-DA rainy, All(L1) | mAP: 66.2 -> 66.9 | Helped |
| Add noise-aware prediction correction (NAPC) (ablation) | Voxel-RCNN · WOD-DA rainy, IoU 0.3 | precision: 31.9 -> 37.1 | Helped |
| Add adaptively weighted instance distillation (AWID) (ablation) | Voxel-RCNN · WOD-DA rainy, IoU 0.3 | recall: 82.0 -> 84.6 | Helped |
Surprising findings
- SPRAY-Aug gives the best All(L1-mAP) on rainy data (66.0 vs 65.9 for DRET-Aug), though DRET-Aug wins on L2 metrics.
- Training SRKD increases training time by 24% (DSVT) to 54% (PV-RCNN++), while inference cost is unchanged.
Limitations
- DRET is two-stage and requires preprocessing to generate particle sets, so it is not end-to-end.
- Training time increases notably, mainly due to the shape similarity computation.
- Rainy evaluation uses only about 3k rainy frames from WOD-DA, and baselines are re-implemented by the authors.
Experimental setup
Datasets: Waymo Open Dataset perception (WOD-P) Waymo Open Dataset domain adaptation (WOD-DA) DRET-simulated rain point clouds
Benchmarks: Waymo Open Dataset WOD-DA rainy subset Waymo Open Dataset WOD-P validation
Metrics: mAP (L1/L2) mAPH (L1/L2) precision recall false positives false negatives
Baselines: Voxel-RCNN PV-RCNN++ DSVT LISA-Aug SPRAY-Aug De-Noising SPG
Builds on: SPRAY LISA OpenPCDet
Closest papers by meaning
- Towards Robust 3D Object Detection with LiDAR and 4D Radar Fusion in Various Weather Conditions (CVPR 2024 · similarity 0.89)
- Equirectangular Point Reconstruction for Domain Adaptive Multimodal 3D Object Detection in Adverse Weather Conditions (AAAI 2025 · similarity 0.87)
- L4DR: LiDAR-4DRadar Fusion for Weather-Robust 3D Object Detection (AAAI 2025 · similarity 0.87)
- An Empirical Study of the Generalization Ability of Lidar 3D Object Detectors to Unseen Domains (CVPR 2024 · similarity 0.87)
- UAWTrack: Universal 3D Single Object Tracking in Adverse Weather (AAAI 2025 · similarity 0.86)
- SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.