Research map / Point clouds / Point cloud classification
Exploring Vision Semantic Prompt for Efficient Point Cloud Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a parameter-efficient fine-tuning paradigm that injects vision semantic prompts from frozen 2D models (CLIP, DINOv2) into frozen 3D point cloud transformers via a Hybrid Attention Adapter, using only about 1.8M trainable parameters.
Problem
Parameter-efficient fine-tuning of pretrained 3D point cloud models performs poorly on complex downstream tasks because frozen backbones yield features with limited semantic cues, causing suboptimal results on real-world and part segmentation tasks.
Approach
Render point clouds into depth maps from three orthogonal views, use frozen 2D pretrained encoders' class tokens (max-pooled and MLP-mapped) as multi-scale vision semantic prompts, and fuse them into 3D tokens via a Hybrid Attention Adapter that modulates 3D features with prompt-generated scale and shift before computing attention.
Key results
- PointMAE + CLIP with the proposed method reaches 89.14% on ScanObjectNN PB-T50-RS (vs. 85.5% for Point-PEFT) and 95.6% on ModelNet40 with voting, using 1.8M trainable parameters.
- With rotation augmentation, the method achieves 90.09% on PB-T50-RS.
- On ShapeNetPart, PointMAE + CLIP achieves 84.70% class mIoU and 86.3% instance mIoU, above the PETL baselines IDPT and DAPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add vision semantic prompts to Point-MAE (ablation) | Point-MAE · ScanObjectNN PB T50 RS | accuracy: 85.18 -> 89.14 | Helped |
| Add a 2D model and concatenate its features with 3D features (ablation) | Point-MAE + DAPT · ScanObjectNN PB T50 RS | accuracy: 85.18 -> 86.95 | Helped |
| Add semantic transfer (ablation) | Point-MAE + DAPT · ScanObjectNN PB T50 RS | accuracy: 86.95 -> 88.02 | Helped |
| Use self-attention with semantically transferred features (ablation) | Point-MAE + DAPT · ScanObjectNN PB T50 RS | accuracy: 88.02 -> 88.26 | Helped |
| Replace self-attention with hybrid attention (ablation) | Point-MAE + DAPT · ScanObjectNN PB T50 RS | accuracy: 88.26 -> 89.14 | Helped |
| Add vision semantic prompts | Point-MAE · ScanObjectNN PB T50 RS | accuracy: 85.18 -> 89.14 | Helped |
| Add vision semantic prompts | Point-MAE · ModelNet40 | accuracy with voting: 93.8 -> 95.6 | Helped |
| Add vision semantic prompts | Point-MAE · ShapeNetPart | Inst. mIoU: 86.1 -> 86.3 | Helped |
| Add vision semantic prompts | Point-MAE · ModelNet40 few-shot classification | accuracy | Mixed |
Limitations
- High FLOPs because of the frozen 2D ViT-B backbones processing three depth maps
- Gains over DINOv2 variant are smaller and results depend on the choice of 2D and 3D backbones
- Paradigm evaluated only with transformer-based 3D and two 2D pretrained models
Experimental setup
Datasets: ScanObjectNN ModelNet40 ShapeNetPart
Benchmarks: ScanObjectNN ModelNet40 ShapeNetPart
Metrics: overall accuracy mIoU trainable parameters FLOPs
Baselines: IDPT DAPT Point-PEFT Point-MAE full fine-tuning Point-BERT full fine-tuning ACT RECon PointMamba
Builds on: Adapter Tuning IDPT DAPT Point-MAE Point-BERT
Closest papers by meaning
- PMA: Towards Parameter-Efficient Point Cloud Understanding via Point Mamba Adapter (CVPR 2025 · similarity 0.88)
- Multi-View Representation is What You Need for Point-Cloud Pre-Training (ICLR 2024 · similarity 0.88)
- GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model (ICML 2025 · similarity 0.88)
- Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models (AAAI 2024 · similarity 0.87)
- Point-PRC: A Prompt Learning Based Regulation Framework for Generalizable Point Cloud Analysis (NeurIPS 2024 · similarity 0.87)
- Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.