Research map / Point clouds / Point cloud classification
GeoAuxNet: Towards Universal 3D Representation Learning for Multi-sensor Point Clouds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GeoAuxNet, which uses hierarchical sensor-aware geometry pools from a voxel-guided dynamic point network as auxiliary point-level supervision to enhance a voxel backbone for jointly training on RGB-D and LiDAR point clouds.
Problem
Point clouds from RGB-D cameras and LiDAR have large domain gaps; point-based methods are accurate but slow, voxel-based methods are efficient but lose fine geometric detail, and existing models are trained separately per sensor.
Approach
Geometry-to-voxel auxiliary learning: a voxel-guided hypernetwork generates point features whose typical local geometric patterns are stored in hierarchical sensor-aware geometry pools, and cross-attention fuses these pool features into voxel features during training; the point branch is dropped at inference.
Key results
- Jointly trained on S3DIS, ScanNet and SemanticKITTI, GeoAuxNet reaches 69.5 mIoU on S3DIS Area 5 test, 71.3 on ScanNet val and 63.8 on SemanticKITTI val, beating jointly trained PPT by about 2 mIoU on each.
- Compared with SPVCNN trained jointly, GeoAuxNet improves mIoU by more than 10 points on each dataset.
- GeoAuxNet runs at 51.8 ms inference (572 instances/sec), versus 80.4 ms for PPT (251/sec), with inference cost close to voxel-only models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add geometry-to-voxel auxiliary learning | MinkowskiNet trained jointly across sensors · S3DIS Area 5 | mIoU: 67.9 -> 69.5; +1.6 | Helped |
| add geometry-to-voxel auxiliary learning | MinkowskiNet trained jointly across sensors · ScanNet Test | mIoU: 69.6 -> 71.3; +1.7 | Helped |
| add geometry-to-voxel auxiliary learning | MinkowskiNet trained jointly across sensors · SemanticKITTI Val | mIoU: 61.8 -> 63.8; +2.0 | Helped |
| share geometry pools across sensors instead of sharing them within each sensor (ablation) | GeoAuxNet · S3DIS Area 5 | mIoU: 69.5 -> 61.4(−8.1) | Hurt |
| use separate geometry pools for each dataset instead of sharing within each sensor (ablation) | GeoAuxNet · S3DIS Area 5 | mIoU: 69.5 -> 67.5(−2.0) | Hurt |
| add voxel features as guidance for the point network (ablation) | GeoAuxNet point network · S3DIS Area 5 | mIoU: 63.3 -> 66.0(+2.7) | Helped |
| add relative point positions (ablation) | GeoAuxNet voxel-guided point network · S3DIS Area 5 | mIoU: 66.0 -> 66.8(+0.8) | Helped |
| add a learnable stage latent code (ablation) | GeoAuxNet voxel-guided point network · S3DIS Area 5 | mIoU: 66.8 -> 67.6(+0.8) | Helped |
Surprising findings
- The original PPT with language-guided categorical alignment fails to converge when trained jointly on RGB-D and LiDAR datasets.
- Sharing geometry pools across sensors drops S3DIS mIoU by 8.1 points, showing sensor-specific pools are needed.
Limitations
- Relies on joint training with sensor-specific embeddings and pools; sharing pools across sensors degrades performance.
- The original PPT fails to converge on joint multi-sensor data, so a modified decoupled-head version was used as baseline.
- Parameter count is matched to about 60M for fairness, and results rely on a fixed set of hyperparameters (threshold, update rate, pool sizes).
Experimental setup
Datasets: S3DIS ScanNet SemanticKITTI ModelNet40
Benchmarks: S3DIS Area 5 ScanNet SemanticKITTI
Metrics: mIoU mAcc inference time (ms) throughput (instances/sec) parameters
Baselines: MinkowskiNet SPVCNN PPT PointTransformer StratifiedTransformer PointNeXt Cylinder3D SphereFormer PointNet++ PTv2
Builds on: MinkowskiNet SPVCNN PPT PointNet++
Closest papers by meaning
- VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection (CVPR 2018 · similarity 0.89)
- Voxel or Pillar: Exploring Efficient Point Cloud Representation for 3D Object Detection (AAAI 2024 · similarity 0.88)
- CurveCloudNet: Processing Point Clouds with 1D Structure (CVPR 2024 · similarity 0.87)
- Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding (CVPR 2024 · similarity 0.86)
- LiDAR-Net: A Real-scanned 3D Point Cloud Dataset for Indoor Scenes (CVPR 2024 · similarity 0.86)
- GeoNLF: Geometry guided Pose-Free Neural LiDAR Fields (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.