Research map / Point clouds / Point cloud classification
Cross-Modal 3D Representation with Multi-View Images and Point Clouds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OpenView, a 3D representation model that fuses multi-view images and point clouds with a cross-modal fusion encoder, sequence-independent modeling, and progressive hard learning, aligned to OpenCLIP's embedding space.
Problem
Existing 3D representation learning uses only point clouds, ignoring the visual detail in multi-view images, which limits cross-modal 3D understanding (retrieval, zero-shot classification, captioning).
Approach
Fuse multi-view image embeddings and point-cloud embeddings with a transformer that uses parallel intra-modal self-attention and inter-modal cross-attention plus self-attention, without positional embeddings for images (order-invariant), trained contrastively against frozen OpenCLIP text embeddings while progressively removing easy images during training.
Key results
- On Objaverse-LVIS, OpenView reaches 52.7% zero-shot Top-1 vs 47.2% for Uni3D (+5.5 points).
- Text-to-3D R@1 on Objaverse-LVIS is 67.6 for OpenView vs 56.1 for OpenShape (about +11.5 points).
- Replacing the cross-modal fusion encoder with a base transformer drops Top-1 zero-shot accuracy on Objaverse-LVIS from 52.7 to 38.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OpenView instead of Uni3D | 3D zero-shot classification · Objaverse-LVIS | Top-1 accuracy: 47.2 -> 52.7 | Helped |
| Use OpenView instead of Uni3D | 3D zero-shot classification · ModelNet40 | Top-1 accuracy: 86.8 -> 87.5 | Helped |
| Use OpenView instead of Uni3D | 3D zero-shot classification · Omniobject3D | Top-1 accuracy: 37.9 -> 58.5 | Helped |
| Use OpenView instead of the strongest listed baseline | 3D retrieval · Objaverse-LVIS Text → 3D | R@1: 56.1 -> 67.6 | Helped |
| Use OpenView instead of the strongest listed baseline | 3D retrieval · Objaverse-LVIS 3D → Text | R@1: 54.0 -> 62.9 | Helped |
| Use OpenView instead of the strongest listed baseline | 3D retrieval · Omniobject3D Text → 3D | R@1: 27.3 -> 41.0 | Helped |
| Use OpenView instead of the strongest listed baseline | 3D retrieval · Omniobject3D 3D → Text | R@1: 22.0 -> 36.8 | Helped |
| add multi-view images (ablation) | OpenView · Objaverse-LVIS Text → 3D | R@1: 56.0 -> 66.2 | Helped |
| add point clouds (ablation) | OpenView · Objaverse-LVIS Text → 3D | R@1: 64.6 -> 66.2 | Helped |
| add the fusion encoder (ablation) | OpenView · Objaverse-LVIS Text → 3D | R@1: 50.0 -> 66.2 | Helped |
| add sequence-independent modeling (ablation) | OpenView · Objaverse-LVIS Text → 3D | R@1: 64.7 -> 66.2 | Helped |
| add inter-modal attention (ablation) | OpenView · Objaverse-LVIS Text → 3D | R@1: 64.0 -> 66.2 | Helped |
Surprising findings
- Removing multi-view images causes a larger drop (>10%) than removing point clouds, suggesting image inputs carry most of the signal for this task.
- OpenView gives only a small advantage on ModelNet40 (87.5 vs 86.8 Top-1 for Uni3D), which the authors attribute to the lack of color and texture in CAD models.
Limitations
- Gap on ModelNet40 is small because CAD models lack color and texture, so multi-view images add little information there.
- Robustness to adversarial attacks through the added image modality is not evaluated.
- Relies on multi-view renderings being available; collection complexity for real-world data is discussed but not measured.
Experimental setup
Datasets: Objaverse ShapeNetCore ABO 3D-Future Objaverse-LVIS ModelNet40 Omniobject3D
Benchmarks: Objaverse-LVIS ModelNet40 Omniobject3D
Metrics: Top-1 Top-3 Top-5 R@1 R@5 R@10
Baselines: PointCLIP v2 ReCon CLIP2Point ULIP OpenShape Uni3D Transformer
Builds on: OpenShape OpenCLIP Point-BERT ULIP curriculum learning
Closest papers by meaning
- Hierarchical Cross-Modal Alignment for Open-Vocabulary 3D Object Detection (AAAI 2025 · similarity 0.88)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.88)
- OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images (NeurIPS 2024 · similarity 0.87)
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.87)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.87)
- CLIP-Driven Open-Vocabulary 3D Scene Graph Generation via Cross-Modality Contrastive Learning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.