Research map / Point clouds / Point cloud classification
Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free GeoZe aggregates per-point CLIP-derived VLM features on the point cloud using local and global geometric (FPFH) and semantic similarity via superpoints, Sinkhorn weighting and anchors, improving zero-shot classification, part and semantic segmentation.
Problem
Zero-shot point cloud understanding via 2D VLMs transfers per-pixel features to 3D points with naive pooling, ignoring the point cloud's geometric structure, which leaves noisy and inconsistent point-level representations.
Approach
A training-free, geometrically driven local-to-global aggregation over superpoints that weights aggregation by FPFH geometric similarity and VLM feature similarity (via Sinkhorn optimal transport), with VLM representation anchors from Mean-Shift to correct drift from the original feature space.
Key results
- GeoZe reaches 70.2% accuracy on ModelNet40, +6.8 over reproduced/reported PointCLIPv2 (repo. 64.2).
- On ShapeNetPart part segmentation GeoZe achieves 57.4 mIoU, +5.6 over PointCLIPv2 (repr. 51.8).
- On ScanNet GeoZe outperforms OpenScene (OpenSeg) by +2.0 mIoU, OpenScene (LSeg) by +3.9 and ConceptFusion by +3.1 on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply geometrically-driven aggregation to transferred VLM representations | PointCLIPv2 on ModelNet40 · ModelNet40 test | accuracy: 64.2 -> 70.2 | Helped |
| apply geometrically-driven aggregation to transferred VLM representations | PointCLIPv2 on ScanObjectNN S-OBJ-ONLY · ScanObjectNN S-OBJ-ONLY test | accuracy: +18.03 | Helped |
| apply geometrically-driven aggregation to transferred VLM representations | PointCLIPv2 on ScanObjectNN S-OBJ-BG · ScanObjectNN S-OBJ-BG test | accuracy: +8.13 | Helped |
| apply geometrically-driven aggregation to transferred VLM representations | PointCLIPv2 on ScanObjectNN S-PB-T50-RS · ScanObjectNN S-PB-T50-RS test | accuracy: +4.37 | Helped |
| apply geometrically-driven aggregation to transferred VLM representations | PointCLIPv2 on ShapeNetPart · ShapeNetPart | mIoU: +5.6 | Helped |
| add superpoint computation and superpoint-to-point aggregation (ablation) | GeoZe on ShapeNetPart · ShapeNetPart | mIoU: +1.7 mIoU | Helped |
| add local aggregation to SC+SP (ablation) | GeoZe on ShapeNetPart · ShapeNetPart | mIoU: 0.5 mIoU | Helped |
| add global aggregation to SC+SP (ablation) | GeoZe on ShapeNetPart · ShapeNetPart | mIoU: 0.8 mIoU | Helped |
| add anchor projection to SC+SP (ablation) | GeoZe on ShapeNetPart · ShapeNetPart | mIoU: 0.8 mIoU | Helped |
| add local and global aggregation to SC+SP (ablation) | GeoZe on ShapeNetPart · ShapeNetPart | mIoU: 57.1 mIoU | Helped |
| include geometric, semantic, and coordinate information in aggregation (ablation) | GeoZe on ShapeNetPart · ShapeNetPart | mIoU: 55.7% vs. 56.9% | Helped |
Surprising findings
- Naive aggregation baseline vs. GeoZe shows only modest gains in some settings, e.g. ShapeNetPart ablation adds 1.7 mIoU from superpoint steps alone.
- GeoZe underperforms on some classes, e.g. aeroplane part segmentation remains hard and several ScanNet and nuScenes classes decrease.
Limitations
- Depends on quality of underlying VLM representations.
- May not handle objects constituting a small portion of the 3D scene.
- Increases inference time compared to naive aggregation.
Experimental setup
Datasets: ModelNet40 ScanObjectNN ShapeNetPart ScanNet nuScenes
Benchmarks: ModelNet40 ScanObjectNN ShapeNetPart ScanNet nuScenes
Metrics: accuracy mIoU
Baselines: PointCLIPv2 OpenScene ConceptFusion PointCLIP
Builds on: CLIP PointCLIPv2 OpenScene ConceptFusion PointNet++
Closest papers by meaning
- Geometry and Uncertainty-Aware 3D Point Cloud Class-Incremental Semantic Segmentation (CVPR 2023 · similarity 0.90)
- Point-MoE: Large-Scale Multi-Dataset Training with Mixture-of-Experts for 3D Semantic Segmentation (ICLR 2026 · similarity 0.90)
- Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition (AAAI 2024 · similarity 0.89)
- Sonata: Self-Supervised Learning of Reliable Point Representations (CVPR 2025 · similarity 0.88)
- Multi-View Representation is What You Need for Point-Cloud Pre-Training (ICLR 2024 · similarity 0.88)
- Generalized Zero-Shot Learning for Point Cloud Segmentation with Evidence-Based Dynamic Calibration (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.