Research map / Point clouds / Point cloud classification
Uni3D: Exploring Unified 3D Representation at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uni3D scales a plain ViT point-cloud encoder to one billion parameters by contrastively aligning point-cloud features with frozen CLIP image/text features, initialized from 2D EVA weights, and reports state-of-the-art zero-shot, few-shot, open-world, and part-segmentation results.
Problem
Scalable, transferable 3D representation learning has been little explored; prior 3D backbones are small or hard to scale, and prior 3D foundation models lack unified architectures and out-of-domain zero-shot capability.
Approach
Use a vanilla ViT (with a PointBERT-style point tokenizer) as a unified 3D encoder, initialize it from strong 2D pretrained weights (EVA), and distill the frozen image-text space of large CLIP teachers into it via multi-modal contrastive alignment, enabling scaling to 1B parameters.
Key results
- Zero-shot ModelNet40 Top1 of 88.2% for Uni3D-g (with LVIS shapes) and 86.8% without LVIS, outperforming OpenShape-PointBERT's 85.3%.
- Objaverse-LVIS zero-shot Top1 rises to 55.3% with the best variant versus 46.8% for OpenShape-PointBERT.
- Few-shot linear probing on Objaverse-LVIS reaches 46.5% at 1-shot and 83.1% at 16-shot, well above OpenShape-PointBERT (24.0% and 51.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale the Uni3D model from Tiny to giant (ablation) | Uni3D · ModelNet40, Ensembled (no LVIS) | Top1 accuracy: 85.8 -> 86.8 | Helped |
| scale the Uni3D model from Tiny to giant (ablation) | Uni3D · Objaverse-LVIS, Ensembled (no LVIS) | Top1 accuracy: 43.5 -> 47.2 | Helped |
| switch to larger CLIP teachers (ablation) | Uni3D · Objaverse-LVIS | Top1 accuracy: 42.3 -> 45.8 | Helped |
| initialize with DINO instead of no initialization (ablation) | Uni3D · Objaverse-LVIS | Top1 accuracy: 44.8 -> 45.0 | Helped |
| initialize with EVA-CLIP instead of no initialization (ablation) | Uni3D · Objaverse-LVIS | Top1 accuracy: 44.8 -> 45.2 | Helped |
| initialize with EVA instead of no initialization (ablation) | Uni3D · Objaverse-LVIS | Top1 accuracy: 44.8 -> 45.8 | Helped |
| freeze the ViT after EVA initialization (ablation) | Uni3D · Objaverse-LVIS | Top1 accuracy: 45.8 -> 15.7 | Hurt |
| add EVA initialization to training with only text features (ablation) | Uni3D · Objaverse-LVIS, only text alignment | Top1 accuracy: 20.7 -> 40.1 | Helped |
| add EVA initialization to training with only image features (ablation) | Uni3D · Objaverse-LVIS, only image alignment | Top1 accuracy: 12.4 -> 26.3 | Helped |
| use Uni3D instead of OpenShape-PointBERT | zero-shot classification · Objaverse-LVIS, Ensembled (no LVIS) | Top1 accuracy: 39.1 -> 47.2 | Helped |
| use Uni3D instead of OpenShape-SparseConv | zero-shot classification · ScanObjectNN, Ensembled (no LVIS) | Top1 accuracy: 54.9 -> 66.5 | Helped |
| use Uni3D instead of CLIP² | zero-shot recognition · ScanNet test | Avg. Top1 accuracy: 38.5 -> 45.8 | Helped |
Surprising findings
- Freezing the EVA-initialized ViT and training only a linear head drops Objaverse-LVIS accuracy to 15.7%, so end-to-end fine-tuning is essential.
- Without EVA initialization, training with only image or only text CLIP features crashes or underperforms, indicating 2D initialization is needed for stable cross-modal training.
Limitations
- Relies on frozen CLIP teachers, so performance is capped by teacher quality and the approach does not learn image or text encoders.
- Real-world ScanNet recognition uses ground-truth instance segmentation, which is given to all methods.
- Part segmentation trains only a feature propagation head on a frozen backbone, and the few-shot setting uses far fewer samples than baselines, so comparisons are not like-for-like.
Experimental setup
Datasets: Objaverse ShapeNet 3D-FUTURE ABO ModelNet40 ScanObjectNN Objaverse-LVIS ScanNet ShapeNetPart
Benchmarks: ModelNet40 ScanObjectNN Objaverse-LVIS ScanNet ShapeNetPart
Metrics: Top1 accuracy Top3 accuracy Top5 accuracy mIoU
Baselines: ULIP-PointBERT OpenShape-PointBERT OpenShape-SparseConv PointCLIP PointCLIP V2 CLIP2Point CLIP^2 PointBERT
Builds on: ULIP OpenShape PointBERT EVA CLIP
Closest papers by meaning
- UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting (ICLR 2025 · similarity 0.90)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.88)
- ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding (CVPR 2024 · similarity 0.88)
- Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent Spaces (CVPR 2025 · similarity 0.88)
- Tango3D: Towards Alignment for Global and Local 2D-3D Correspondence (NeurIPS 2026 · similarity 0.88)
- Unifying 2D and 3D Vision-Language Understanding (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.