Research map / Point clouds / Point cloud classification
OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OpenDlign, which fine-tunes a small transformer block on OpenCLIP ViT-H-14 to align multi-view depth maps with diffusion-generated depth-aligned images from ShapeNet, achieving SOTA zero-shot and few-shot 3D classification.
Problem
Open-world 3D representation learning relies on CAD-rendered images with limited texture diversity and on extra 3D encoders trained on small 3D datasets, which fails to fully transfer CLIP's knowledge to 3D; depth-map-based approaches also suffer from noisy contours and RGB-biased prompts.
Approach
Replace CAD-rendered images with diffusion-generated, depth-conditioned (ControlNet depth) images that have diverse textures, align depth maps of a contour-aware projection to them via residual fine-tuning of one transformer block on CLIP's image encoder, use depth-specific prompts, and aggregate logits from pre-trained and fine-tuned encoders.
Key results
- Zero-shot Top1 on ModelNet40 reaches 82.6% for OpenDlign (ViT-H-14), 8.0 points above the prior best TAMM-SparseConv (74.6%) trained on ShapeNet.
- Zero-shot Top1 on OmniObject3D reaches 31.3%, 16.4 points above prior ShapeNet-trained methods.
- Adding depth-aligned images to OpenShape-SparseConv and TAMM-PointBERT improves their ShapeNet-only results on the benchmarks, e.g., TAMM-PointBERT (+dlign) reaches 57.3% Top1 on ScanObjectNN versus 54.8% for TAMM-PointBERT, a gain of 2.5 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OpenDlign instead of the strongest ShapeNet-trained baseline | zero-shot classification · ModelNet40 | Top1 accuracy: 74.6 -> 82.6 | Helped |
| use OpenDlign instead of the strongest ShapeNet-trained baseline | zero-shot classification · ScanObjectNN | Top1 accuracy: 57.9 -> 59.5 | Helped |
| use OpenDlign instead of the strongest ShapeNet-trained baseline | zero-shot classification · OmniObject3D | Top1 accuracy: 14.9 -> 31.3 | Helped |
| use OpenDlign instead of the strongest ShapeNet-trained baseline | zero-shot classification · Objaverse-LVIS | Top1 accuracy: 13.7 -> 35.0 | Helped |
| add contour-aware projection (ablation) | pre-trained CLIP for zero-shot classification · ModelNet40 | Top1 accuracy: 59.7 -> 68.8; +9.1 | Helped |
| add multimodal alignment to contour-aware projection (ablation) | OpenDlign · ModelNet40 | Top1 accuracy: 68.8 -> 79.2 | Helped |
| align depth maps with depth-aligned images rather than CAD-rendered images (ablation) | multimodal alignment · ScanObjectNN | Top1 accuracy: 52.7 -> 59.5 | Helped |
| pre-align depth-aligned images with text before depth-dlign alignment (ablation) | multimodal alignment · ScanObjectNN | Top1 accuracy: 51.1 -> 59.5 | Hurt |
| add depth-specific text prompts (ablation) | OpenDlign zero-shot classification · ModelNet40 | Top1 accuracy: 59.7 -> 75.9 | Helped |
| add multi-view logit aggregation (ablation) | OpenDlign zero-shot classification · ModelNet40 | Top1 accuracy: 79.2 -> 81.0 | Helped |
| use an independent residual trainable block instead of directly tuning the encoder (ablation) | OpenDlign fine-tuning · ModelNet40 | Top1 accuracy: 56.8 -> 82.2 | Helped |
| increase trainable block count from 1 to 2 (ablation) | OpenDlign fine-tuning · ModelNet40 | Top1 accuracy: 82.6 -> 82.3 | No clear effect |
Surprising findings
- Aligning depth maps with CAD-rendered images performs worse than aligning with depth-aligned images, by 6.8% on ScanObjectNN.
- Aligning depth-aligned images with text before depth alignment hurts performance (lowest ablation results).
Limitations
- Cannot generate depth-aligned images for the largest 3D dataset (about 1M objects) due to limited compute.
- Requires data filtering because some depth-aligned images are low quality.
- Training billion-parameter 3D open-world models with these images is too expensive.
Experimental setup
Datasets: ShapeNet ModelNet40 ScanObjectNN OmniObject3D Objaverse-LVIS ScanNet V2 DFN5B
Benchmarks: ModelNet40 ScanObjectNN OmniObject3D Objaverse-LVIS ScanNet V2
Metrics: Top1 accuracy Top3 accuracy Top5 accuracy mAP at IoU 0.25 mAP at IoU 0.5
Baselines: PointCLIP PointCLIP V2 CLIP2Point ULIP OpenShape TAMM OV-3DET CoDA
Builds on: CLIP PointCLIP V2 OpenShape TAMM ControlNet
Closest papers by meaning
- Occlusion-aware Text-Image-Point Cloud Pretraining for Open-World 3D Object Recognition (CVPR 2025 · similarity 0.88)
- Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition (AAAI 2024 · similarity 0.88)
- Cross-Modal 3D Representation with Multi-View Images and Point Clouds (CVPR 2025 · similarity 0.87)
- UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting (ICLR 2025 · similarity 0.87)
- More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding (AAAI 2025 · similarity 0.86)
- Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.