Research map / Point clouds / Segmentation and detection
Multi-Modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a scene-level weakly supervised 3D point cloud segmentation method that infers multi-modality (geometry-only and RGB-appended) point affinities and normalizes classifier weights to mitigate long-tailed bias.
Problem
Scene-level weakly supervised semantic segmentation of RGB-D point clouds fails to fully exploit geometric and appearance modalities, point affinity from feature networks, and suffers from long-tailed class distributions.
Approach
A two-stream shared-parameter PointNet++ computes affinity matrices from RGB-masked and RGB-appended point clouds across multiple SA scales; classifier weight normalization (NCW) is applied to reduce head-class bias in affinity, and the refined affinity smooths logits for pseudo-labeling in a MIL-plus-self-training framework.
Key results
- On ScanNet validation, MMA reaches 37.7% mIoU versus 29.6% for WyPR, and 30.6% on the test set versus WyPR's 24%.
- On S3DIS area 5, MMA achieves 26.3% mIoU versus 22.3% for WyPR.
- Ablation: normalizing classifier weights combined with multi-modality affinity raises mIoU from 28.4% (A.3) to 36.2% (A.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MMA instead of WyPR | Scene-level weakly supervised 3D segmentation · ScanNet val | mIoU: 31.1 -> 37.7 | Helped |
| Use MMA instead of WyPR | Scene-level weakly supervised 3D segmentation · ScanNet test | mIoU: 24 -> 30.6 | Helped |
| Use MMA instead of WyPR | Scene-level weakly supervised 3D segmentation · S3DIS area 5 test | mIoU: 22.3 -> 26.3 | Helped |
| Add normalized classifier weights (NCW) to the baseline (ablation) | MIL-based segmentation baseline · ScanNet val | mIoU: 23.8 -> 26.9 | Helped |
| Add multi-modality affinity to the baseline (ablation) | MIL-based segmentation baseline · ScanNet val | mIoU: 23.8 -> 28.4 | Helped |
| Add multi-modality affinity and NCW to the baseline (ablation) | MIL-based segmentation baseline · ScanNet val | mIoU: 28.4 -> 36.2 | Helped |
| Add multi-scale affinity to the baseline (ablation) | MIL-based segmentation baseline · ScanNet val | mIoU: 23.8 -> 25.7 | Helped |
| Add multi-modality and multi-scale affinity to the baseline (ablation) | MIL-based segmentation baseline · ScanNet val | mIoU: 23.8 -> 30.1 | Helped |
| Remove the Transformer block from the student head (ablation) | Full MMA · ScanNet val | mIoU: 37.7 -> 33.9 | Hurt |
| Add Fᵐ₁ features to Fᵐᵤ for affinity (ablation) | Multi-modality affinity · ScanNet val | mIoU: 36.2 -> 36.6 | Helped |
| Add Fᵐ₂ features to affinity already using Fᵐᵤ and Fᵐ₁ (ablation) | Multi-modality affinity · ScanNet val | mIoU: 36.6 -> 37.7 | Helped |
| Add NCW to MMA (ablation) | MMA affinity · ScanNet tail classes | mAP: 65.7 -> 72.9; +7.2 | Helped |
Surprising findings
- Normalizing classifier weights alone gives only a modest gain (23.8% to 26.9%), but combined with the multi-modality affinity it yields a much larger improvement, suggesting the two components are strongly interdependent.
Limitations
- Only ScanNet and S3DIS indoor benchmarks are evaluated.
- Long-tail groups are defined by ad hoc point-count thresholds on the training split.
Experimental setup
Datasets: ScanNet S3DIS
Benchmarks: ScanNet S3DIS
Metrics: mIoU mAP
Baselines: MPRM WyPR MIL-Derived PCAM PointNet++ PointCNN KPConv MinkNet
Builds on: PointNet++ Decoupling Representation and Classifier multiple instance learning
Closest papers by meaning
- Weakly Supervised Semantic Point Cloud Segmentation: Towards 10x Fewer Labels (CVPR 2020 · similarity 0.92)
- Weakly Supervised Point Cloud Semantic Segmentation via Artificial Oracle (CVPR 2024 · similarity 0.90)
- Less Is More: Label Recommendation for Weakly Supervised Point Cloud Semantic Segmentation (AAAI 2024 · similarity 0.89)
- Rethinking Few-shot 3D Point Cloud Semantic Segmentation (CVPR 2024 · similarity 0.88)
- Density-Guided Semi-Supervised 3D Semantic Segmentation with Dual-Space Hardness Sampling (CVPR 2024 · similarity 0.87)
- Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.