Research map / Point clouds / Point cloud classification
Sonata: Self-Supervised Learning of Reliable Point Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sonata builds a self-distillation point SSL framework on Point Transformer V3 that avoids a "geometric shortcut" via encoder-only training, feature up-casting, and progressive curricula, pretrained on 140k scenes, yielding strong linear probing on 3D perception.
Problem
Existing 3D point cloud self-supervised models collapse to low-level geometric cues (normals, height), so their representations are weak under linear probing and not reliable for diverse 3D tasks.
Approach
Identify the "geometric shortcut" caused by point coordinates entering operators, and counter it by removing the decoder so SSL runs on coarse encoder features, obscuring spatial info of masked points, up-casting multi-scale features, and scheduling task difficulty, within a DINOv2-style self-distillation framework.
Key results
- Linear probing on ScanNet semantic segmentation reaches 72.5% mIoU, versus 21.8% for the prior MSC method and 63.1% for DINOv2 aggregated features.
- Full fine-tuning achieves 79.4% mIoU on ScanNet and 82.3% on S3DIS 6-fold, state of the art among the compared methods.
- With 1% of ScanNet scenes, Sonata full fine-tuning reaches 45.3% mIoU versus 29.2% for MSC, and linear probing reaches 43.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MSC pretraining with Sonata | PTv3 linear probing · ScanNet Val semantic segmentation | mIoU: 21.8 -> 72.5 | Helped |
| pretrain with Sonata instead of DINOv2 features | linear probing · ScanNet Val semantic segmentation | mIoU: 63.09 -> 72.52 | Helped |
| combine Sonata and DINOv2 features | linear probing · ScanNet Val semantic segmentation | mIoU: 72.52 -> 75.91 | Helped |
| use Sonata features rather than training from scratch with 1% of scenes | ScanNet semantic segmentation · ScanNet, 1% scenes | mIoU: 25.8 -> 45.3 | Helped |
| remove the hierarchical decoder during self-supervised learning (ablation) | Sonata · ScanNet semantic segmentation | mIoU: 20.7% to 60.4% | Helped |
| add feature up-casting once (ablation) | Sonata without decoder · ScanNet semantic segmentation | mIoU: 60.4 -> 62.6 | Helped |
| add feature up-casting twice (ablation) | Sonata without decoder · ScanNet semantic segmentation | mIoU: 62.6 -> 64.4 | Helped |
| increase feature up-casting from two to three times (ablation) | Sonata · ScanNet semantic segmentation | mIoU: 64.4 -> 64.2 | Hurt |
| fine-tune the full Sonata model rather than linear-probe it | Sonata · ScanNet Val semantic segmentation | mIoU: 72.5 -> 79.4 | Helped |
| use linear probing rather than full fine-tuning | Sonata · ScanNet200 Val semantic segmentation | mIoU: 36.8 -> 29.3 | Hurt |
Surprising findings
- Linear probing of Sonata surpasses supervised PTv3 on 1% of ScanNet data, and its linear probe outperforms the DINOv2 features projected onto point clouds.
- Removing the hierarchical decoder alone raised linear probing from 20.7% to 60.4%.
Limitations
- Linear probing still underperforms full fine-tuning and performs poorly on large-vocabulary tasks such as ScanNet200 and ScanNet++ (100 classes).
- Representations struggle to distinguish a large number of classes.
Experimental setup
Datasets: ScanNet ScanNet++ S3DIS ArkitScenes HM3D Structured3D ASE nuScenes Waymo SemanticKITTI
Benchmarks: ScanNet ScanNet200 ScanNet++ S3DIS nuScenes Waymo SemanticKITTI
Metrics: mIoU mAP mAcc allAcc
Baselines: PointContrast CSC MSC PPT PTv3 DINOv2 SparseUNet
Builds on: DINOv2 Masked Scene Contrast Point Transformer V3 PPT
Closest papers by meaning
- Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds (CVPR 2026 · similarity 0.90)
- LinNet: Linear Network for Efficient Point Cloud Representation Learning (NeurIPS 2024 · similarity 0.89)
- Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding (CVPR 2024 · similarity 0.88)
- Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations (NeurIPS 2025 · similarity 0.88)
- MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding (AAAI 2024 · similarity 0.88)
- Point-Cache: Test-time Dynamic and Hierarchical Cache for Robust and Generalizable Point Cloud Analysis (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.