Research map / Vision-language and multimodal / CLIP and zero-shot
ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ULIP-2, a tri-modal 3D pre-training framework that auto-generates holistic language descriptions for rendered views with BLIP-2 (no manual 3D annotations), aligning point clouds, images and text in frozen OpenCLIP space.
Problem
Multimodal 3D pre-training relies on manually annotated or metadata-derived language descriptions that are not scalable and lack diversity and detail.
Approach
Render each 3D shape from holistic viewpoints, caption each view with BLIP-2 (10 samples, keep the CLIP-top-1), and align 3D point cloud features to frozen pre-aligned OpenCLIP image and text features, with scaled-up CLIP (ViT-G) and 3D backbones.
Key results
- Zero-shot top-1 of 50.6% on Objaverse-LVIS and 84.7% on ModelNet40 with Point-BERT pre-trained on Objaverse(no LVIS)+ShapeNet, surpassing OpenShape's 46.8% on Objaverse-LVIS.
- ScanObjectNN hardest set overall accuracy 91.5% with PointNeXt (1.4M params), up from 87.5% from scratch.
- 3D captioning CIDEr improves from 132.2 (ULIP) to 160.5 (ULIP-2) with X-InstructBLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ULIP-2 instead of OpenShape with Objaverse + ShapeNet pre-training | Point-BERT · Objaverse-LVIS, zero-shot | top-1 accuracy: 46.8 -> 50.6 | Helped |
| use ULIP-2 instead of OpenShape with Objaverse + ShapeNet pre-training | Point-BERT · ModelNet40, zero-shot | top-1 accuracy: 84.4 -> 84.7 | Helped |
| pre-train with ULIP-2 instead of training PointNeXt from scratch | PointNeXt · ScanObjectNN hardest set | overall accuracy: 87.5 -> 91.1 | Helped |
| use ULIP-2-pre-trained Point-BERT rather than ULIP-pre-trained Point-BERT | X-InstructBLIP · 3D-to-language generation | CIDEr: 132.2 -> 160.5 | Helped |
| replace manual captions with top-1 holistic BLIP-2 captions (ablation) | Point-BERT pre-trained on ShapeNet · ModelNet40, zero-shot | top-1 accuracy: 60.4 -> 69.7 | Helped |
| use BLIP-2 rather than BLIP to generate captions (ablation) | Point-BERT pre-trained on ShapeNet · ModelNet40, zero-shot | top-1 accuracy: 67.7 -> 69.7 | Helped |
| increase holistic views from 1 to 30 (ablation) | Point-BERT pre-trained on ShapeNet · ModelNet40, zero-shot | top-1 accuracy: 54.8 -> 69.7 | Helped |
| select the top-1 rather than top-3 CLIP-ranked captions per view (ablation) | Point-BERT pre-trained on ShapeNet · ModelNet40, zero-shot | top-1 accuracy: 66.7 -> 69.7 | Helped |
| scale the CLIP backbone from ViT-B to ViT-G (ablation) | Point-BERT pre-trained on Objaverse without LVIS · ModelNet40, zero-shot | top-1 accuracy: 71.4 -> 76.3 | Helped |
| scale the 3D encoder from 21.9M to 32.5M parameters (ablation) | Point-BERT with ViT-G · ModelNet40, zero-shot | top-1 accuracy: 76.3 -> 77.0 | Helped |
| scale the 3D encoder from 32.5M to 43.1M parameters (ablation) | Point-BERT with ViT-G · ModelNet40, zero-shot | top-1 accuracy: 77.0 -> 76.8 | Hurt |
Surprising findings
- Replacing manual captions with BLIP-2 holistic captions alone raises ModelNet40 zero-shot top-1 from 60.4% to 69.7%.
- Using only the top-1 CLIP-ranked caption per view outperforms ensembling top-3, top-5 or top-10 captions.
Limitations
- Pre-training is limited to object-level 3D shape datasets, not scene-level data.
- Language descriptions depend on the quality of the underlying large multimodal model (BLIP-2 vs BLIP ablation).
Experimental setup
Datasets: Objaverse ShapeNet ModelNet40 Objaverse-LVIS ScanObjectNN
Benchmarks: Objaverse-LVIS ModelNet40 ScanObjectNN
Metrics: top-1 accuracy top-5 accuracy overall accuracy class-average accuracy CIDEr
Baselines: ULIP OpenShape ReCon CLIP2Point PointCLIP PointCLIPv2 Point-MAE PointNeXt
Builds on: ULIP CLIP BLIP-2 X-InstructBLIP Point-BERT
Closest papers by meaning
- TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understanding (CVPR 2024 · similarity 0.89)
- ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding (NeurIPS 2025 · similarity 0.89)
- UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting (ICLR 2025 · similarity 0.88)
- Uni3D: Exploring Unified 3D Representation at Scale (ICLR 2024 · similarity 0.88)
- More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding (AAAI 2025 · similarity 0.87)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.