Research map / Point clouds / Point cloud classification
Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MixCon3D, which fuses multi-view image features with point cloud features into a holistic 3D object representation aligned to text in CLIP-based image-text-3D contrastive pre-training, along with an improved training recipe.
Problem
Existing contrastive language-image-3D pre-training methods align point cloud features to image and text embeddings separately, ignoring complementary multi-view and geometric information of 3D objects, and their training recipes are underexplored.
Approach
Concatenate multi-view-pooled image features with point cloud features and pass them through a learnable 3D sculptor to form a 3D object-level representation contrastively aligned with text, combined with an improved training recipe (large batch, separate temperatures, cosine LR with warmup, EMA).
Key results
- On Objaverse-LVIS with PointBERT and Ensemble training, MixCon3D reaches 52.5% zero-shot top-1 vs. 46.8% for OpenShape (+5.7%).
- On ScanObjectNN, MixCon3D reaches 58.6% top-1 vs. 52.2% for OpenShape (+6.4%).
- The improved training recipe raises the OpenShape baseline by 3.3%, 3.6%, and 1.9% top-1 on Objaverse-LVIS, ScanObjectNN, and ModelNet40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the separate temperature strategy (ablation) | Image-text-3D contrastive training recipe · ScanObjectNN | Top1: 52.0 -> 52.8 | Helped |
| Increase the batch size (ablation) | Image-text-3D contrastive training recipe · Objaverse-LVIS | Top1: 46.8 -> 48.0 | Helped |
| Use cosine learning-rate decay with warmup (ablation) | Image-text-3D contrastive training recipe · Objaverse-LVIS | Top1: 48.0 -> 48.5 | Helped |
| Add exponential moving average (ablation) | Image-text-3D contrastive training recipe · ScanObjectNN | Top1: 54.1 -> 55.6 | Helped |
| Add image-text alignment loss (ablation) | MixCon3D · Objaverse-LVIS | Top1: 49.8 -> 48.7 | Hurt |
| Add 3D-text alignment loss (ablation) | MixCon3D · Objaverse-LVIS | Top1: 49.8 -> 51.0 | Helped |
| Add multi-view image features (ablation) | MixCon3D with both image-text and 3D-text losses · Objaverse-LVIS | Top1: 51.6 -> 52.5 | Helped |
| Add multi-view image features (ablation) | MixCon3D with both image-text and 3D-text losses · ScanObjectNN | Top1: 58.1 -> 58.6 | Helped |
| Replace no multi-view fusion with view-pooling (ablation) | MixCon3D · Objaverse-LVIS | Top1: 51.6 -> 52.5 | Helped |
| Add a fully connected layer after view-pooling (ablation) | MixCon3D · ScanObjectNN | Top1: 58.6 -> 52.4 | Hurt |
| Increase the number of views during training (ablation) | MixCon3D · Objaverse-LVIS | Top1: 51.6 -> 53.2 | Helped |
| Increase the number of views from 4 to 8 (ablation) | MixCon3D · ScanObjectNN | Top1: 0.5% | Hurt |
Surprising findings
- Adding image-text contrastive loss alone (without the 3D-text term) slightly degrades performance relative to the baseline on Objaverse-LVIS and ModelNet40.
- Adding a fully connected layer after view-pooling improves Objaverse-LVIS slightly but drops ScanObjectNN top-1 by 6.2%.
Limitations
- Multi-view image features are only usable when images are available at inference; point-cloud-only inference performs worse.
- Increasing the number of views improves Objaverse-LVIS but can slightly hurt ScanObjectNN performance.
Experimental setup
Datasets: ShapeNet 3D-FUTURE ABO Objaverse ModelNet40 ScanObjectNN Objaverse-LVIS
Benchmarks: Objaverse-LVIS ScanObjectNN ModelNet40
Metrics: top-1 accuracy top-1 class average accuracy top-3 accuracy top-5 accuracy
Baselines: ULIP OpenShape PointCLIP PointCLIP v2 ReCon CG3D CLIP2Point
Builds on: ULIP OpenShape CLIP Point-BERT
Closest papers by meaning
- Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding (AAAI 2024 · similarity 0.88)
- More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding (AAAI 2025 · similarity 0.87)
- Uni3D: Exploring Unified 3D Representation at Scale (ICLR 2024 · similarity 0.87)
- CLIP-Driven Open-Vocabulary 3D Scene Graph Generation via Cross-Modality Contrastive Learning (CVPR 2024 · similarity 0.86)
- MM-Mixing: Multi-Modal Mixing Alignment for 3D Understanding (AAAI 2025 · similarity 0.86)
- RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.