Research map / Point clouds / Shape and surface completion
Duoduo CLIP: Efficient 3D Understanding with Multi-View Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Duoduo CLIP trains a multi-view image encoder, initialized from OpenCLIP ViT-B/32 with cross-view attention, to align 3D shapes with CLIP text/image space, matching or beating point-cloud methods at a fraction of the compute.
Problem
Existing 3D-text representation methods rely on point clouds, which have limited resolution, a domain gap with real images, poor use of 2D CLIP priors, and high training cost; they also require full scans of objects.
Approach
Replace the point-cloud shape encoder in contrastive 3D-text alignment with a CLIP ViT over multi-view renders, averaging per-view [CLS] embeddings and adding multi-view attention (tokens from all views attend jointly) in the last layers, while freezing early layers and MLPs.
Key results
- On Objaverse-LVIS (Ensembled), Duoduo CLIP with 12 frames reaches 55.2 top-1, on par with or above Uni3D (55.3 top-1 with 1B params) while using a much smaller model.
- On MVPNet, Duoduo CLIP (+MVImgNet, 12F) reaches 66.06 top-1 versus 54.59 for OpenShape (+MVPNet).
- On Text2Shape retrieval, Duoduo CLIP achieves RR@1 15.90 and RR@5 34.08, above OpenShape (10.53/25.94) and Uni3D-G (10.78/26.37) trained on Ensembled.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-view attention instead of omitting it (ablation) | Duoduo CLIP with 6 trainable layers · Objaverse-LVIS | Top1 accuracy: 54.61 -> 55.24 (+0.63) | Helped |
| Use multi-view attention instead of omitting it (ablation) | Duoduo CLIP with 6 trainable layers · MVPNet | Top1 accuracy: 47.87 -> 49.16 | Helped |
| Use multi-view attention instead of omitting it (ablation) | Duoduo CLIP with 6 trainable layers · ScanObjectNN | Top1 accuracy: 58.77 -> 66.32 | Helped |
| Train only 3 MVA layers instead of 6 (ablation) | Duoduo CLIP · Objaverse-LVIS | Top1 accuracy: 55.24 -> 53.77 | Hurt |
| Train all 12 layers instead of the default 6 (ablation) | Duoduo CLIP · Objaverse-LVIS | Top1 accuracy: 55.24 -> 55.32 | Helped |
| Increase training-view sampling from [1, 4] to [1, 6] (ablation) | Duoduo CLIP · Objaverse-LVIS | Top1 accuracy: +0.24 | Helped |
| Increase training-view sampling from [1, 6] to [1, 8] (ablation) | Duoduo CLIP · Objaverse-LVIS | Top1 accuracy: +0.09 | Helped |
| Also fine-tune MLP layers after attention (ablation) | Duoduo CLIP with 6 trainable attention layers · Objaverse-LVIS | Top1 accuracy: 55.24 -> 55.96 | Helped |
| Also fine-tune MLP layers after attention (ablation) | Duoduo CLIP with 6 trainable attention layers · MVPNet | Top1 accuracy: 49.16 -> 45.46 | Hurt |
| Also fine-tune MLP layers after attention (ablation) | Duoduo CLIP with 6 trainable attention layers · ScanObjectNN | Top1 accuracy: 66.32 -> 59.63 | Hurt |
| Initialize the shape encoder from OpenAI CLIP rather than the same OpenCLIP checkpoint as frozen encoders (ablation) | Duoduo CLIP (6 layers) · Objaverse-LVIS | Top1 accuracy: 55.2 -> 53.0 | Hurt |
| Initialize the shape encoder from OpenAI CLIP rather than the same OpenCLIP checkpoint as frozen encoders (ablation) | Duoduo CLIP (6 layers) · ScanObjectNN | Top1 accuracy: 66.3 -> 60.8 | Hurt |
Surprising findings
- Zero-shot OpenCLIP BigG-14 already beats OpenShape on the no-LVIS Objaverse-LVIS split, suggesting point-cloud methods generalize worse than CLIP.
- On ScanNet, the zero-shot B-32 CLIP model is the best performer (69.2 avg), outperforming all fine-tuned models.
Limitations
- Multi-view images are potentially weaker than point clouds for geometric understanding.
- Only CLIP-style contrastive pretraining is explored; overfitting appears when training too many layers.
- Performance on MVPNet and ScanNet is limited by dataset size; fine-tuned models underperform zero-shot CLIP on some real-world data.
Experimental setup
Datasets: Objaverse ABO ShapeNet 3D-FUTURE MVImgNet MVPNet ScanObjectNN ScanNet Text2Shape Zero123 renderings
Benchmarks: Objaverse-LVIS ScanObjectNN MVPNet ScanNet Text2Shape
Metrics: top-1 accuracy top-3 accuracy top-5 accuracy retrieval rate at 1 retrieval rate at 5 inference time
Baselines: OpenShape Uni3D ULIP TAMM MixCon3D ShapeLLM VIT-LENS Text2Shape Parts2Words TriCoLo MXM-CLR ZS B-32
Builds on: OpenShape CLIP MVDream Zero123
Closest papers by meaning
- Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition (AAAI 2024 · similarity 0.88)
- Multi-View Representation is What You Need for Point-Cloud Pre-Training (ICLR 2024 · similarity 0.88)
- Occlusion-aware Text-Image-Point Cloud Pretraining for Open-World 3D Object Recognition (CVPR 2025 · similarity 0.88)
- Learning Partonomic 3D Reconstruction from Image Collections (CVPR 2025 · similarity 0.87)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.87)
- Uni3D: Exploring Unified 3D Representation at Scale (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.