Research map / Vision-language and multimodal / Human and embodied modeling
Hierarchical Intra-modal Correlation Learning for Label-free 3D Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a hierarchical intra-modal correlation learning framework (intra-set label refinement, intra-scene and inter-scene attention, feedback distillation) that improves label-free 3D semantic segmentation trained from MaskCLIP pseudo-labels, reaching 36.6% mIoU on ScanNet and 23.0% on nuScenes.
Problem
Label-free 3D semantic segmentation distills noisy and inconsistent pseudo-labels from pre-trained vision-language models (MaskCLIP/CLIP) into 3D networks, which hurts the learned 3D feature space.
Approach
Model intra-modal (3D point) correlations at three levels: refine 2D pseudo-labels by majority voting within geometric consistency sets, use transformer attention within each scene and across scenes to pull visually and geometrically similar point features together, and distill the aggregated features back into the 3D model via a feedback KL loss.
Key results
- Achieves 36.6% mIoU on ScanNet, 7.8 points above CLIP2Scene (28.8 on the ScanNet validation ablation baseline; 25.6 in Table 1).
- Achieves 23.0% mIoU on nuScenes, 2.2 points above CLIP2Scene (20.8).
- Ablation on ScanNet validation: intra-set refinement +2.3, intra-scene +3.3 (cumulative), inter-scene +7.3 cumulative, feedback distillation brings final 36.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the hierarchical intra-modal correlation learning framework | CLIP2Scene on ScanNet · ScanNet | mIoU: 25.6 -> 36.6 | Helped |
| add the hierarchical intra-modal correlation learning framework | CLIP2Scene on nuScenes · nuScenes | mIoU: +2.2% mIoU | Helped |
| use Ours instead of Chen et al. | label-free 3D segmentation · nuScenes | mIoU: 26.8 -> 23.0 | Hurt |
| add intra-set label refinement (ablation) | CLIP2Scene baseline · ScanNet validation | mIoU: 28.8 -> 31.1 (+2.3) | Helped |
| add intra-scene correlation learning (ablation) | intra-set-refined model · ScanNet validation | mIoU: 31.1 -> 32.1 | Helped |
| add inter-scene correlation learning (ablation) | intra-set- and intra-scene model · ScanNet validation | mIoU: 32.1 -> 36.1 | Helped |
| add feedback distillation (ablation) | model with intra-set, intra-scene, and inter-scene learning · ScanNet validation | mIoU: 36.1 -> 36.6 | Helped |
| train for 120 rather than 100 epochs (ablation) | Ours on ScanNet · ScanNet | mIoU: +1.4% mIoU | Helped |
| train for 120 rather than 100 epochs (ablation) | CLIP2Scene on ScanNet · ScanNet | mIoU: +0.3% mIoU | Helped |
Surprising findings
- Gains on nuScenes are smaller than on ScanNet because CLIP pseudo-labels on nuScenes are so poor that erroneous labels dominate, limiting label refinement.
- Method reaches 36.6% mIoU with lower training cost (14 GPU-hours) than Chen et al. (20 GPU-hours) and without SAM.
Limitations
- Intra-set refinement cannot correct erroneous pseudo-labels when CLIP labels are mostly wrong, as on nuScenes.
- Intra-modal correlations within images and texts are not explored.
- Sampling limited to M=1024 points per batch to manage GPU memory.
Experimental setup
Datasets: ScanNet nuScenes
Benchmarks: ScanNet nuScenes
Metrics: mIoU training GPU hours parameter count
Baselines: MaskCLIP MaskCLIP+ OpenScene CLIP2Scene Chen et al. (SAM-based label refinement)
Builds on: CLIP2Scene MaskCLIP Segment Anything
Closest papers by meaning
- Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception (CVPR 2024 · similarity 0.88)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.87)
- AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correction (AAAI 2025 · similarity 0.87)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.87)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.86)
- LabelAny3D: Label Any Object 3D in the Wild (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.