Research map / Detection and segmentation / 3D detection and driving
AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaCo, a label-free 3D semantic segmentation method that corrects noisy VFM-derived point labels with adaptive timing, historical-prediction voting, cluster-based label propagation, and an adaptive robust loss.
Problem
Label-free 3D semantic segmentation of outdoor LiDAR point clouds using visual foundation model pseudo labels, which are noisy due to calibration errors, occlusion, illumination, and unknown objects.
Approach
Combines a cross-modal label generation module (FastSAM masks + SSA-Engine descriptions + word2vec matching, unprojected and voted across frames), an adaptive noise corrector that fits per-sample mIoU learning curves to choose a correction time and relabels using historical prediction modes and DBSCAN cluster voting, and an adaptive robust loss mixing CE, Lovasz, NCE and MAE terms.
Key results
- On nuScenes, AdaCo reaches 31.2 mIoU versus 26.8 for TowardsVFMs (the strongest label-free method in the table, +4.4).
- On SemanticKITTI validation, AdaCo reaches 25.7 mIoU, a 17.6-point gain over the MaskCLIP-supervised baseline (8.1), and 12.5 and 11.5 points above the unsupervised GrowSp and U3DS.
- Ablation on SemanticKITTI validation: CLGM alone gives 19.5 mIoU; adding ANC gives 25.3; adding ARL alone gives 21.4; full model gives 25.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the reproduced MaskCLIP baseline with AdaCo | CMDFusion · nuScenes val | mIoU (%): 16.5 -> 31.2 | Helped |
| Replace the reproduced MaskCLIP baseline with AdaCo | CMDFusion · SemanticKITTI validation | mIoU (%): 8.1 -> 25.7 | Helped |
| Add CLGM to the baseline (ablation) | CMDFusion · SemanticKITTI validation | mIoU (%): 8.1 -> 19.5 | Helped |
| Add ANC to CLGM (ablation) | AdaCo · SemanticKITTI validation | mIoU (%): 19.5 -> 25.3 | Helped |
| Add ARL to CLGM (ablation) | AdaCo · SemanticKITTI validation | mIoU (%): 19.5 -> 21.4 | Helped |
| Increase adjacent-frame setting from 1 to 2 (ablation) | CLGM · SemanticKITTI validation | mIoU (%): 25.4 -> 25.9 | Helped |
| Increase history sequence length from 3 to 5 (ablation) | ANC · SemanticKITTI validation | mIoU (%): 18.5 -> 25.3 | Helped |
| Correct once per sample rather than each time the derivative starts to decrease (ablation) | ANC · SemanticKITTI validation | mIoU (%): 18.9 -> 24.0 | Helped |
| Use word2vec semantic similarity instead of a manual dictionary (ablation) | CLGM · nuScenes label quality | mIoU (%): 17.6 -> 30.4 | Helped |
| Use word2vec semantic similarity instead of a manual dictionary (ablation) | CLGM · SemanticKITTI label quality | mIoU (%): 18.9 -> 27.9 | Helped |
| Set derivative threshold to 0.9 rather than 0.5 or 0.99 (ablation) | ANC · SemanticKITTI validation | mIoU (%): 0.99: 22.5; 0.5: 19.5; 0.9: 25.7 | Helped |
| Set winner portion ω to 3 rather than 2 or 4 (ablation) | ANC · SemanticKITTI validation | mIoU (%): 2: 23.4; 3: 25.7; 4: 19.0 | Helped |
Surprising findings
- Correcting labels only once per sample performed better (24.0 mIoU) than correcting each time the derivative decreased (18.9 mIoU).
- Replacing a manual keyword dictionary with word2vec similarity raised label quality substantially (nuScenes 17.6 to 30.4 mIoU, SemanticKITTI 18.9 to 27.9).
Limitations
- Depends on pixel-to-point calibration for label transfer.
- Limited by VFM semantic understanding in driving scenes; may fail on small image targets and diverse categories.
- Hyperparameters (derivative threshold, winner portion, sigma, history length) selected empirically with limited sensitivity analysis.
Experimental setup
Datasets: SemanticKITTI nuScenes
Benchmarks: SemanticKITTI nuScenes
Metrics: mIoU mAcc
Baselines: MaskCLIP MaskCLIP++ CLIP2Scene OpenScene TowardsVFMs Hierarchical GrowSp U3DS
Builds on: CMDFusion MaskCLIP PNAL FastSAM Lovasz loss
Closest papers by meaning
- Towards Label-free Scene Understanding by Vision Foundation Models (NeurIPS 2023 · similarity 0.88)
- Hierarchical Intra-modal Correlation Learning for Label-free 3D Semantic Segmentation (CVPR 2024 · similarity 0.87)
- A²LC: Active and Automated Label Correction for Semantic Segmentation (AAAI 2026 · similarity 0.87)
- Active Label Correction for Semantic Segmentation with Foundation Models (ICML 2024 · similarity 0.86)
- UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior (NeurIPS 2024 · similarity 0.85)
- Omni-Query Active Learning for Source-Free Domain Adaptive Cross-Modality 3D Semantic Segmentation (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.