Research map / Point clouds / Segmentation and detection
3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AFOV, a two-stage annotation-free 3D segmentation framework that distills masks and labels from 2D open-vocabulary segmentation models (SAN, FC-CLIP, CAT-Seg) into a point cloud backbone via tri-modal contrastive pre-training and an Approximate Flat Interaction post-processing module.
Problem
Point cloud annotation is expensive in autonomous driving; existing annotation-free 3D methods based on CLIP or SAM are noisy or lack text correspondence, so the paper seeks better 3D representation learning and annotation-free segmentation without labels.
Approach
Use masks, mask labels, mask features and text features synchronously generated by CLIP-based 2D open-vocabulary segmentation models; pre-train a 3D backbone with superpixel-superpoint and text-superpoint contrastive losses (TMP), then train with pseudo-labels projected via LiDAR-camera calibration, and refine predictions with a non-parametric direction-based interaction module (AFI).
Key results
- Annotation-free 3D segmentation on nuScenes val reaches 47.73% mIoU with SAN teacher, +3.13% over the previous best (OV3D 44.60%).
- 1% fine-tuning on nuScenes reaches 51.75% mIoU with AFOV+SAN, and 48.14% mIoU on SemanticKITTI with AFOV-TMP+CAT-Seg.
- 100% linear probing on nuScenes reaches 56.35% mIoU, +4.81% over the prior best pre-training method (VLM2Scene 51.54%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AFOV with SAN instead of OV3D | nuScenes annotation-free segmentation · nuScenes val | mIoU: 44.60 -> 47.73 | Helped |
| Add TMP to the annotation-free baseline (ablation) | AFOV with SAN · nuScenes annotation-free segmentation | mIoU: 44.16 -> 47.42 | Helped |
| Add AFI to the annotation-free baseline (ablation) | AFOV with FC-CLIP, CAT-Seg, or SAN · nuScenes annotation-free segmentation | mIoU: 39.00 -> 41.89; 38.45 -> 41.79; 44.16 -> 46.26 | Helped |
| Add both TMP and AFI (ablation) | AFOV with SAN · nuScenes annotation-free segmentation | mIoU: 44.16 -> 47.73 | Helped |
| Use a class dictionary to map open-vocabulary prompts to dataset classes (ablation) | AFOV with CAT-Seg · nuScenes annotation-free segmentation | mIoU: 32.31 -> 38.45 | Helped |
| Add text-superpoint contrastive loss to TMP (ablation) | AFOV with CAT-Seg · nuScenes annotation-free segmentation | mIoU: 39.84 -> 40.92 | Helped |
| Use superpixel-superpoint groups instead of k-NN point groups (ablation) | AFOV with CAT-Seg TMP · nuScenes annotation-free segmentation | mIoU: 39.34 -> 40.92 | Helped |
| Use superpixel-superpoint groups instead of k-NN point groups (ablation) | AFOV with CAT-Seg TMP · nuScenes, 1% fine-tuning | mIoU: +4.74% mIoU | Helped |
| Use CAT-Seg mask features rather than a ResNet50 image backbone (ablation) | AFOV with CAT-Seg TMP · nuScenes annotation-free segmentation | mIoU: 39.84 -> 40.92 | Helped |
| Pre-generate 2D features and remove the 2D backbone during TMP (ablation) | AFOV-TMP with CAT-Seg · nuScenes annotation-free segmentation | mIoU: 40.12 -> 40.92 | Helped |
| Pre-generate 2D features and remove the 2D backbone during TMP (ablation) | AFOV-TMP with CAT-Seg · Pre-training | time per epoch: 3.44 -> 2.10 | Helped |
| Add image inference to AFOV with TMP and AFI (ablation) | AFOV with CAT-Seg · nuScenes annotation-free segmentation | mIoU: 42.83 -> 43.64 | Helped |
Surprising findings
- Teacher choice matters more than the framework: SAN-based baseline outperforms FC-CLIP and CAT-Seg by over 5%, and FC-CLIP/CAT-Seg variants with TMP and AFI do not beat the SAN baseline.
- Pre-generating 2D features without the 2D backbone during TMP gave comparable or better results than keeping a 2D backbone, at lower pre-training time.
Limitations
- Image assistance gives limited improvement (about 0.5% mIoU on average).
- Annotation-free training relies on pseudo-labels, leading to relatively weak point cloud understanding.
- The class dictionary and AFI are specific to outdoor point clouds and may not generalize to other settings.
Experimental setup
Datasets: nuScenes SemanticKITTI COCO
Benchmarks: nuScenes SemanticKITTI COCO
Metrics: mIoU accuracy
Baselines: CLIP2Scene Chen et al. (NeurIPS'23) OpenScene OV3D SLidR ST-SLidR SEAL VLM2Scene
Builds on: SLidR Point-NN CLIP OpenScene
Closest papers by meaning
- 3D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentation (CVPR 2025 · similarity 0.87)
- VLM2Scene: Self-Supervised Image-Text-LiDAR Learning with Foundation Models for Autonomous Driving Scene Understanding (AAAI 2024 · similarity 0.86)
- OpenBox: Annotate Any Bounding Boxes in 3D (NeurIPS 2025 · similarity 0.86)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.86)
- MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving (AAAI 2024 · similarity 0.86)
- JiSAM: Alleviate Labeling Burden and Corner Case Problems in Autonomous Driving via Minimal Real-World Data (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.