Research map / Point clouds / Segmentation and detection
Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MM-FSS, a cost-free multimodal few-shot 3D point cloud segmentation model that uses textual class names explicitly and 2D image features implicitly (via LSeg-aligned pretraining), with MCF/MSF fusion and a test-time adaptive calibration (TACC).
Problem
Few-shot 3D point cloud semantic segmentation relies only on unimodal point clouds, ignoring readily available modalities such as class names and paired 2D images.
Approach
Use a shared Stratified Transformer backbone with an intermodal head (aligned to LSeg 2D features during pretraining) and a unimodal head, fuse their prototype correlations (MCF), refine them with LSeg text-embedding semantic guidance weighted per point and class (MSF), and calibrate test predictions with an episode-adaptive IoU-based indicator (TACC).
Key results
- On S3DIS, MM-FSS reaches 52.09 mean mIoU in 1-way 1-shot (+4.3 over COSeg) and 50.16 in 2-way 5-shot (+8.7).
- On ScanNet, MM-FSS reaches 44.73 mean mIoU in 1-way 1-shot (+2.7) and 39.21 in 2-way 1-shot (+10.2 over COSeg).
- Ablation on ScanNet: adding TACC raises 1-shot mIoU from 42.83 to 44.73 and 5-shot from 48.04 to 50.07.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MM-FSS instead of COSeg† | few-shot point cloud segmentation · S3DIS and ScanNet, all 1/2-way and 1/5-shot settings | mIoU (%): +4.53% and +8.58% on S3DIS 1/2-way; +3.41% and +9.92% on ScanNet 1/2-way | Helped |
| Add MCF to MM-FSS without fusion modules (ablation) | MM-FSS · ScanNet 1-way 1-shot | mIoU (%): 40.69 -> 41.45 | Helped |
| Add MSF to MM-FSS without fusion modules (ablation) | MM-FSS · ScanNet 1-way 1-shot | mIoU (%): 40.69 -> 42.21 | Helped |
| Combine MCF and MSF (ablation) | MM-FSS · ScanNet 1-way 1-shot and 1-way 5-shot | mIoU (%): 42.21 -> 42.83; 46.46 -> 48.04 | Helped |
| Add the UF head alongside the IF head (ablation) | MM-FSS · ScanNet 1-way 1-shot | mIoU (%): 35.10 -> 42.83 | Helped |
| Add TACC (ablation) | MM-FSS with IF and UF heads · ScanNet 1-way 1-shot and 1-way 5-shot | mIoU (%): 42.83 -> 44.73; 48.04 -> 50.07 | Helped |
| Increase MSF blocks from 3 to 5 (ablation) | MM-FSS without TACC · ScanNet 1-way 1-shot | mIoU (%): 43.33 -> 44.69 | Helped |
| Add image modality to the 3D-only model, then add text modality (ablation) | MM-FSS · ScanNet 1-way 1-shot | mIoU (%): 40.69 -> 41.45 -> 44.73 | Helped |
| Replace adaptive TACC coefficient with fixed 1:1 or 1:0.5 coefficients (ablation) | MM-FSS · ScanNet 1-way 1-shot | mIoU (%): 44.73 vs 43.76; 44.73 vs 43.24 | Hurt |
| Use dynamic point-category weighting instead of linear MSF weighting (ablation) | MM-FSS · ScanNet 1-way 1-shot and 1-way 5-shot | mIoU (%): 40.80 -> 42.83; 45.79 -> 48.04 | Helped |
| Use OpenSeg instead of LSeg for pretraining (ablation) | MM-FSS · ScanNet few-shot settings | mIoU (%): 43.47 vs 44.73 (1-way 1-shot mean); 47.09 vs 45.48 (1-way 5-shot split 1) | Mixed |
| Aggregate 5-shot adaptive indicators with min instead of mean (ablation) | MM-FSS TACC · ScanNet 1-way 5-shot, split 0 | mIoU (%): 55.46 -> 54.00 | Hurt |
Surprising findings
- COSeg retrained with the same 2D-aligned pretrained backbone (COSeg†) gives little improvement over COSeg, suggesting the fusion modules matter more than the backbone weights.
Limitations
- May learn inductive bias toward the studied datasets; generalization to other scenarios is untested.
- Requires GPU training with associated carbon emissions.
Experimental setup
Datasets: S3DIS ScanNet
Benchmarks: S3DIS ScanNet
Metrics: mIoU
Baselines: AttMPTI QGE QGPA COSeg COSeg†
Builds on: COSeg Stratified Transformer LSeg KPConv
Closest papers by meaning
- Rethinking Few-shot 3D Point Cloud Semantic Segmentation (CVPR 2024 · similarity 0.92)
- Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model (CVPR 2025 · similarity 0.89)
- Text Augmented Correlation Transformer For Few-shot Classification & Segmentation (CVPR 2025 · similarity 0.88)
- Multi-Modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation (AAAI 2024 · similarity 0.86)
- Taylor Series-Inspired Local Structure Fitting Network for Few-shot Point Cloud Semantic Segmentation (AAAI 2025 · similarity 0.86)
- MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.