Research map / Point clouds / Segmentation and detection
Frozen CLIP Transformer Is an Efficient Point Cloud Encoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EPCL, which keeps a CLIP ViT transformer frozen and trains only a point cloud tokenizer, a task token and task heads, to act as an efficient point cloud encoder for detection, segmentation, classification and few-shot learning.
Problem
Pretraining strong 3D point cloud models is hard because of scarce 3D data and the domain gap between point clouds and images; the paper asks whether a frozen 2D CLIP transformer can serve as a point cloud encoder.
Approach
Freeze the CLIP image transformer and learn only a point cloud tokenizer (FPS + KNN + MLP), a learnable task token and task-specific heads, so point tokens land in a token space similar to image tokens, with no paired 2D-3D data and no 3D pretraining.
Key results
- Frozen CLIP + CAGroup3D reaches 61.1 AP50 on ScanNet V2 detection versus 60.8 for CAGroup3D, and 43.0 AP50 with 3DETR head versus 42.1 for MaskPoint.
- Reports 71.5 mIoU on S3DIS Area5, above MaskPoint (67.1) and Simple3D-Former (67.0).
- Achieves 92.9% OA on ModelNet40 classification, versus 93.8 for MaskPoint and 93.2 for PointBERT; trainable parameters are about 9.1% on S3DIS per the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use frozen CLIP with a 3DETR head instead of the MaskPoint baseline | ScanNet V2 detection · ScanNet V2 detection | AP50: 42.1 -> 43.0 | Helped |
| Use frozen CLIP with a CAGroup3D head instead of CAGroup3D | ScanNet V2 detection · ScanNet V2 detection | AP50: 60.8 -> 61.1 | Helped |
| Use EPCL instead of Point Transformer | S3DIS Area5 semantic segmentation · S3DIS Area5 | mIoU: 70.4 -> 71.5 | Helped |
| Use EPCL instead of RPVNet | SemanticKITTI validation segmentation · SemanticKITTI validation | mIoU: 69.6 -> 72.4 | Helped |
| Use EPCL instead of MaskPoint | ModelNet40 classification · ModelNet40 | OA: 93.8 -> 92.9 | Hurt |
| Add the task token (ablation) | EPCL on ScanNet V2 detection · ScanNet V2 | AP50: 59.2 -> 61.1 | Helped |
| Keep CLIP frozen during training (ablation) | EPCL on ScanNet V2 detection · ScanNet V2 | AP50: 60.1 -> 61.1 | Helped |
| Keep CLIP frozen during training (ablation) | EPCL on ModelNet40 classification · ModelNet40 | OA: 92.3 -> 92.9 | Helped |
| Replace frozen CLIP with frozen SAM (ablation) | EPCL on ScanNet V2 detection · ScanNet V2 | AP50: 59.5 -> 61.1 | Hurt |
| Replace frozen CLIP with frozen DINO (ablation) | EPCL on ScanNet V2 detection · ScanNet V2 | AP50: 60.0 -> 61.1 | Hurt |
| Use task token instead of prompts at every transformer layer (ablation) | Frozen CLIP classification · ModelNet40 classification | accuracy: 93.03 -> 92.91 | Hurt |
| Use task token instead of prompts at every transformer layer (ablation) | Frozen CLIP detection · ScanNet V2 detection | AP50: 42.43 -> 43.00 | Helped |
Surprising findings
- Unfreezing the CLIP transformer lowers ScanNet V2 detection AP50 by 1.0 point, which the authors attribute to small 3D datasets degrading the pretrained parameter space.
- Using the frozen CLIP transformer with 2D-only CLIP surpasses SAM and DINO frozen encoders on ScanNet V2 detection.
Limitations
- Gains on some benchmarks are small, e.g. 61.1 vs 60.8 AP50 with CAGroup3D and 92.9 vs 93.8 OA on ModelNet40 against MaskPoint.
- Several results depend on task-specific heads (3DETR, CAGroup3D, segmentation decoders) and on the tokenizer being fine-tuned, so the frozen encoder alone is not the whole method.
- Evidence for 2D-3D alignment is qualitative (saliency maps and cross-correlation on a few categories).
Experimental setup
Datasets: ScanNet V2 S3DIS SemanticKITTI ModelNet40 ShapeNet
Benchmarks: ScanNet V2 S3DIS (Area5) SemanticKITTI ModelNet40 ScanObjectNN
Metrics: AP50 AP25 mIoU mAcc OA few-shot accuracy
Baselines: PointBERT MaskPoint TAP Simple3D-Former SoftGroup CAGroup3D Point Transformer Cylinder3D PVKD 2DPASS RPVNet RangeFormer
Builds on: CLIP Vision Transformer PointBERT 3DETR CAGroup3D
Closest papers by meaning
- Point Transformer V3: Simpler Faster Stronger (CVPR 2024 · similarity 0.87)
- Multi-View Representation is What You Need for Point-Cloud Pre-Training (ICLR 2024 · similarity 0.87)
- Exploring Vision Semantic Prompt for Efficient Point Cloud Understanding (ICML 2025 · similarity 0.87)
- Point Cloud Pre-training with Diffusion Models (CVPR 2024 · similarity 0.87)
- On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation (NeurIPS 2025 · similarity 0.86)
- Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.