Research map / Point clouds / Point cloud classification
On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GEM, a geometry-aware PEFT module (spatial adapter with 3D sparse convolution on positional encodings plus a context adapter using a few learned latent tokens attending to all points) that matches or exceeds full fine-tuning of point cloud transformers on 3D scene segmentation while tuning about 1.6% of parameters.
Problem
Parameter-efficient fine-tuning of large pre-trained point cloud transformers for large-scale 3D scene segmentation performs poorly with existing PEFT methods, which treat points as orderless tokens and ignore local spatial structure and global scene context under local attention.
Approach
GEM combines a Spatial Adapter (lightweight 3D convolutional bottleneck refining the pre-trained positional encoding over local voxel neighborhoods) with a Context Adapter (a few learned latent tokens that attend to all points, giving O(nm) global context with m much smaller than n, shared across blocks).
Key results
- On ScanNet Val with Sonata backbone, GEM reaches 78.3 mIoU using 1.6% learnable parameters, matching Sonata full fine-tuning (79.4 mIoU) and beating LoRA (76.7) and Adapter (77.0).
- On ScanNet++ Val, GEM achieves 46.6 mIoU versus 43.7 for Sonata full fine-tuning.
- On outdoor Semantic KITTI transfer from indoor pre-training, GEM reaches 67.7 mIoU versus 63.8 for the best PEFT baseline (LoRA), though below the outdoor-pretrained full fine-tuning (72.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GEM instead of full fine-tuning | Sonata · ScanNet Val | mIoU: 78.3 -> 78.3 | No clear effect |
| Add GEM instead of full fine-tuning | Sonata · ScanNet200 Val | mIoU: 37.3 -> 35.6 | Hurt |
| Add GEM instead of full fine-tuning | Sonata · ScanNet++ Val | mIoU: 49.8 -> 46.6 | Hurt |
| Add GEM instead of LoRA | Sonata · S3DIS Area 5 | mIoU: 74.5 -> 75.1 | Helped |
| Add GEM instead of LoRA | Sonata · Sem.KITTI Val | mIoU: 63.8 -> 67.7 | Helped |
| Add GEM | PTv3-PPT · ScanNet Val | mIoU: 78.6 -> 79.1 | Helped |
| Add GEM | Sonata with decoder · ScanNet Val | mIoU: 79.1 -> 79.5 | Helped |
| Add the Context Adapter to the Spatial Adapter (ablation) | GEM on Sonata · ScanNet Val | mIoU: 77.2 -> 78.3 | Helped |
| Add the Spatial Adapter to the Context Adapter (ablation) | GEM on Sonata · ScanNet Val | mIoU: 77.3 -> 78.3 | Helped |
| Set the latent token count to 1 instead of 4 (ablation) | GEM on Sonata · ScanNet Val | mIoU: 78.3 -> 78.1 | No clear effect |
| Set the latent token count to 8 instead of 4 (ablation) | GEM on Sonata · ScanNet Val | mIoU: 78.3 -> 78.2 | No clear effect |
| Share latent tokens globally rather than keeping them layer-local (ablation) | GEM on Sonata · ScanNet Val | mIoU: 77.8 -> 78.3 | Helped |
Surprising findings
- A single shared latent token in the context adapter already achieves 78.1 mIoU, suggesting one token suffices for global scene context.
- Prompt tuning underperforms linear probing on S3DIS 6-fold, and LoRA with rank 1 collapses to 42.5 mIoU on ScanNet.
Limitations
- Relies on parallelizable computation; large batch sizes incur noticeable overhead due to sequential processing.
- Gradients must back-propagate through early layers, which can hinder training efficiency on dense point clouds such as S3DIS.
- Outdoor transfer from indoor pre-training still leaves a noticeable gap to training from scratch on outdoor data.
Experimental setup
Datasets: ScanNet ScanNet200 ScanNet++ S3DIS Semantic KITTI ShapeNetPart
Benchmarks: ScanNet ScanNet200 ScanNet++ S3DIS Semantic KITTI ShapeNetPart
Metrics: mIoU mAcc allAcc
Baselines: BitFit Adapter LoRA Prompt Tuning Linear probing (Sonata lin.) Sonata full fine-tuning PTv3 SparseUNet PointLoRA PointGST
Builds on: Sonata Point Transformer V3 LoRA adapter tuning prompt tuning
Closest papers by meaning
- Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models (AAAI 2024 · similarity 0.93)
- GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model (ICML 2025 · similarity 0.90)
- S$^2$AM3D: Scale-controllable Part Segmentation of 3D Point Clouds (CVPR 2026 · similarity 0.89)
- Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding (CVPR 2024 · similarity 0.87)
- LinNet: Linear Network for Efficient Point Cloud Representation Learning (NeurIPS 2024 · similarity 0.87)
- A Linear Time and Space Local Point Cloud Geometry Encoder via Vectorized Kernel Mixture (VecKM) (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.