Research map / 3D vision / Face and shape
PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PartDistill, a bi-directional cross-modal distillation framework that transfers 2D knowledge from VLMs (CLIP or GLIP) into a 3D point/mesh student network for zero-shot and few-shot 3D shape part segmentation.
Problem
Zero-shot 3D shape part segmentation using 2D vision-language models suffers from uncovered 3D regions, inconsistent/incorrect 2D VLM predictions, and no knowledge accumulation across different 3D shapes.
Approach
A teacher VLM provides 2D part predictions back-projected to 3D; a point cloud student (Point-M2AE encoder plus distillation head) learns them via masked confidence-weighted cross-entropy across many shapes, and a backward distillation step re-scores each 2D knowledge unit by teacher–student agreement so low-quality units are suppressed.
Key results
- On ShapeNetPart zero-shot, the pre-alignment version reaches 63.9% overall mIoU with CLIP (vs. 48.4% PointCLIPv2) and 64.1% with GLIP, claimed 15.5% higher than the best competitor.
- On PartNetE zero-shot, the TTA version reaches 39.9% overall mIoU vs. 27.3% for PartSLIP (12.6% higher).
- On PartNetE 8-shot, the method reaches 65.9% overall mIoU vs. 59.4% for PartSLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill GLIP predictions into a 3D student rather than transfer boxes directly (ablation) | PartDistill · ShapeNetPart Chair | mIoU (%): 60.1 -> 73.5 | Helped |
| add pre-alignment across a collection of shapes to forward distillation (ablation) | PartDistill · ShapeNetPart Chair | mIoU (%): 73.5 -> 84.2 | Helped |
| add backward distillation to pre-aligned forward distillation (ablation) | PartDistill · ShapeNetPart Chair | mIoU (%): 84.2 -> 86.5 | Helped |
| exclude predictions for uncovered points (ablation) | PartDistill · ShapeNetPart Chair | mIoU (%): 86.5 -> 80.4 | Hurt |
| initialize Point-M2AE weights randomly and make them learnable instead of using frozen pretrained weights (ablation) | PartDistill · ShapeNetPart Chair | mIoU (%): 86.5 -> 86.7 | No clear effect |
| add generated shapes to train-set distillation data (ablation) | PartDistill · ShapeNetPart Chair | mIoU (%): 86.2 -> 88.4 | Helped |
| use generated data alone instead of train-set data (ablation) | PartDistill · ShapeNetPart Chair | mIoU (%): 86.2 -> 85.3 | Hurt |
| use CLIP instead of GLIP as the VLM with backward distillation (ablation) | PartDistill · ShapeNetPart Chair | mIoU (%): 86.5 -> 65.0 | Hurt |
| increase the number of rendered views from fewer than 6 to 6 or more (ablation) | PartDistill · ShapeNetPart | mIoU | Helped |
| use sparse gray point clouds instead of mesh or dense colored point clouds for rendering (ablation) | PartDistill · ShapeNetPart | mIoU | Hurt |
Surprising findings
- Using generated shapes from DiT-3D alone gives competitive ShapeNetPart results compared with training-set distillation (e.g., 69.0 vs 69.3 for airplane).
Limitations
- Segmentation quality is bounded by the VLM's predictions, which are trained on object-level rather than part-level categories.
- Only a single VLM is used; combining multiple VLMs is left to future work.
Experimental setup
Datasets: ShapeNetPart PartNetE ShapeNet55 COSEG
Benchmarks: ShapeNetPart PartNetE COSEG
Metrics: mIoU
Baselines: PointCLIP PointCLIPv2 OpenScene PartSLIP SATR PointNet++ PointNext ACD Prototype Point-M2AE
Builds on: PointCLIP PointCLIPv2 PartSLIP SATR Point-M2AE OpenScene
Closest papers by meaning
- PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understanding (NeurIPS 2025 · similarity 0.89)
- PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models (CVPR 2025 · similarity 0.87)
- Learning Partonomic 3D Reconstruction from Image Collections (CVPR 2025 · similarity 0.87)
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding (NeurIPS 2025 · similarity 0.86)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.