Research map / Point clouds / Segmentation and detection
JiSAM: Alleviate Labeling Burden and Corner Case Problems in Autonomous Driving via Minimal Real-World Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JiSAM, a plug-and-play method that jointly trains a LiDAR 3D detector on CARLA simulation point clouds and about 2.5% of NuScenes labels, using jittering augmentation, a domain-aware input layer, and memory-based sectorized feature alignment.
Problem
LiDAR 3D object detection for autonomous driving relies on heavy real-world labeling, and corner-case categories are missing from real data; naively adding simulator data hurts performance because of low sample efficiency and the sim-to-real gap.
Approach
Adds Gaussian noise in spherical coordinates to simulation point clouds for sample efficiency, uses separate input layers per domain (sharing the rest of the backbone), and aligns simulation BEV object features to a momentum-updated memory bank of real features indexed by shape-context sector and heading bin.
Key results
- With about 2.5% real labels plus CARLA data, JiSAM on TransFusion reaches 63.95 mAP and 69.36 NDS, close to full-label TransFusion (64.51 mAP, 69.31 NDS).
- Compared with TransFusion trained on the same 2.5% real labels, JiSAM gains roughly 4 mAP and 3 NDS.
- With motorcycle labels removed from real data, JiSAM reaches about 16 AP on motorcycle while staying within 0.5 AP on car and pedestrian.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add synthetic data without JiSAM components (ablation) | Transfusion trained with 2.5% NuScenes labels · NuScenes | mAP: 60.18 -> 58.29 (-1.89) | Hurt |
| Add domain-specific backbone (ablation) | Transfusion trained with synthetic data and 2.5% NuScenes labels · NuScenes | mAP: 60.18 -> 62.30 (+2.12) | Helped |
| Add memory-based sectorized alignment loss (ablation) | Transfusion with domain-specific backbone · NuScenes | mAP: 62.30 -> 63.71 | Helped |
| Add jittering augmentation (ablation) | Transfusion with domain-specific backbone · NuScenes | mAP: 62.30 -> 63.28 | Helped |
| Use 50% synthetic data with jittering augmentation (ablation) | Transfusion with JiSAM components · NuScenes | mAP: 63.90 vs. 63.95 | No clear effect |
| Train with JiSAM and 2.5% real labels plus simulation data | Transfusion · NuScenes | mAP: approximately 4 mAP | Helped |
| Train with JiSAM and simulation data while omitting real motorcycle labels | Transfusion · NuScenes motorcycle corner case | AP: around 16% APs | Helped |
Surprising findings
- Naively adding simulation data to the 2.5% real-label training lowers mAP (60.18 to 58.29) and NDS, showing that simulation data hurts without the proposed components.
- Jittering augmentation with 50% of the simulation data matches using all simulation data without jittering, indicating better sample efficiency.
Limitations
- Motorcycle and bicycle performance remains below full-label training, attributed to limited CARLA asset diversity and sparse points.
- Evaluated only on NuScenes with CARLA as the simulator, so generality to other datasets and simulators is not shown beyond a brief claim of plug-and-play use.
Experimental setup
Datasets: NuScenes CARLA simulation LiDAR point clouds
Benchmarks: NuScenes
Metrics: mAP NDS per-category AP
Baselines: TransFusion trained on 100% real labels TransFusion trained on 2.5% real labels VoxelNeXt CenterPoint SECOND PointPillars
Builds on: TransFusion OpenPCDet CARLA
Closest papers by meaning
- Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data (CVPR 2022 · similarity 0.89)
- LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World (CVPR 2020 · similarity 0.89)
- NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields (AAAI 2024 · similarity 0.87)
- Accurate Training Data for Occupancy Map Prediction in Automated Driving Using Evidence Theory (CVPR 2024 · similarity 0.86)
- 3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous Driving (AAAI 2025 · similarity 0.85)
- The Road Ahead in Autonomous Driving: The KITScenes Multimodal Dataset (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.