Research map / Point clouds / Segmentation and detection
Multi-View Representation is What You Need for Point-Cloud Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MVNet, a point-cloud pre-training method that distills hierarchical features of pre-trained 2D models (DINOv2 ViT-B) into a 3D sparse U-Net via projected feature maps, plus a multi-view correspondence loss to retain 3D signal.
Problem
Point-cloud pre-training suffers from scarce 3D data and lack of annotations, and existing methods (mostly contrastive or synthetic-object reconstruction) give limited transfer gains to real-scene and shape-level downstream tasks.
Approach
Train a 3D feature network directly and align its projected 2D feature maps with the hierarchical features of a frozen pre-trained 2D ViT (2D knowledge transfer loss), while a cross-attention transformer predicts cross-view pixel correspondences to keep 3D geometry in the features (multi-view consistency loss).
Key results
- Semantic segmentation: MVNet reaches 75.6 val mIoU on ScanNet and 73.8 mIoU on S3DIS Area 5, about +1.5 and +1.6 over the second-best pre-training method.
- 3D detection with CAGroup3D on ScanNet reaches 76.0 mAP@0.25 and 62.4 mAP@0.5.
- Shape-level ScanObjectNN PB-T50-RS accuracy rises to 87.8 with MVNet-B (Objaverse) and 91.0 with MVNet-L; ShapeNetPart ins. mIoU reaches 86.2/86.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the 2D knowledge transfer loss (ablation) | MVNet without the multi-view consistency loss · ScanNet | val mIoU: 72.2 -> 74.7 | Helped |
| Add the multi-view consistency loss (ablation) | MVNet without the 2D knowledge transfer loss · ScanNet | val mIoU: 72.2 -> 73.6 | Helped |
| Combine 2D knowledge transfer and multi-view consistency losses (ablation) | MVNet · ScanNet | val mIoU: 74.7 -> 75.6 | Helped |
| Set pre-training masking ratio to 30% (ablation) | MVNet · ScanNet | val mIoU: 74.8 -> 75.6 | Helped |
| Set pre-training masking ratio to 60% rather than 30% (ablation) | MVNet · ScanNet | val mIoU: 75.6 -> 74.2 | Hurt |
| Set pre-training masking ratio to 80% rather than 30% (ablation) | MVNet · ScanNet | val mIoU: 75.6 -> 73.5 | Hurt |
| Use view overlap ratios of 40% to 80% instead of 20% (ablation) | MVNet with multi-view consistency · ScanNet | val mIoU: 74.7 -> 75.1; 75.3; 75.4 | Helped |
| Increase candidate second views from 1 to 5 (ablation) | MVNet multi-view consistency module · ScanNet | multi-view consistency loss: 72.9 -> 73.6 | Helped |
| Increase candidate second views from 1 to 5 (ablation) | MVNet 2D knowledge transfer module · ScanNet | 2D knowledge transfer loss: 74.5 -> 74.7 | No clear effect |
| Use DINOv2-B instead of other tested 2D pre-trained models (ablation) | MVNet · ScanNet | semantic segmentation val mIoU | Helped |
| Use larger X-B models instead of smaller X-S models (ablation) | MVNet 2D knowledge transfer · ScanNet | semantic segmentation val mIoU | Helped |
| Start applying the 2D knowledge transfer loss at the first or second layer instead of the third (ablation) | MVNet · ScanNet | semantic segmentation val mIoU | Hurt |
Surprising findings
- Transferring only encoder pre-trained weights, not the decoder, gave better segmentation fine-tuning performance.
- Adding the 2D knowledge transfer loss from the first or second layer hurt performance, while starting from the third layer was best.
Limitations
- Authors note plans to extend to outdoor settings, implying indoor-only scope so far.
- Scaling the feature encoding network and adding more pre-training data (e.g., NYUv2) are left for future work.
- Performance on the ScanNet ablation depends on the choice of 2D pre-trained model; DINOv2 was best among those tested.
Experimental setup
Datasets: ScanNet Objaverse ShapeNet S3DIS SUN RGB-D ShapeNetPart ScanObjectNN
Benchmarks: ScanNet S3DIS SUN RGB-D ShapeNetPart ScanObjectNN
Metrics: mIoU overall accuracy mean accuracy mAP@0.25 mAP@0.5 instance mIoU class mIoU accuracy
Baselines: Jigsaw OcCo STRL MM-3DScene PointContrast DepthContrast SceneContext RandomRooms 4DContrast Point-BERT MaskPoint PointBERT
Builds on: PointContrast DINOv2 SR-UNet VoteNet CAGroup3D
Closest papers by meaning
- Exploring Vision Semantic Prompt for Efficient Point Cloud Understanding (ICML 2025 · similarity 0.88)
- Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding (CVPR 2024 · similarity 0.88)
- Point Cloud Pre-training with Diffusion Models (CVPR 2024 · similarity 0.88)
- Duoduo CLIP: Efficient 3D Understanding with Multi-View Images (ICLR 2025 · similarity 0.88)
- Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition (AAAI 2024 · similarity 0.88)
- Interpretable3D: An Ad-Hoc Interpretable Classifier for 3D Point Clouds (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.