Research map / Point clouds / Point cloud classification
How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that 3D point cloud transformers (PTv3, Sonata, SplatFormer, SpatialLM) are highly token-redundant and proposes gitmerge3D, a globally informed graph-based token merging method that cuts tokens by 80-95% with small accuracy loss across segmentation, reconstruction and detection.
Problem
Point cloud transformers such as PTv3 use dense token representations, incurring high computation and memory costs during training and inference.
Approach
A 3D-specific token merging scheme that scores each token by its mean negative cosine similarity to all patch centroids (a globally informed energy score), adaptively merges low-energy patches aggressively and high-energy patches moderately, and unmerges to restore per-point features for dense prediction.
Key results
- On PTv3 with ScanNet, merging 90% of tokens drops GFLOPs from 107.5 to 19.9 and memory from 10.12 GB to 1.6 GB (per intro), with mIoU staying close to baseline (77.4 at 0.8 merge rate fine-tuned vs 77.6 baseline).
- On nuScenes with PTv3 + Ours, peak memory falls from 6.20 GB to 0.92 GB and GFLOPS from 101.68 to 32.45, while mIoU drops from 80.3 to 78.0.
- Random Drop, FPS, and VoxelGrid downsampling degrade OOD novel view synthesis markedly at 80% token reduction, while the proposed method stays within about 0.1% of SplatFormer across metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add token merging at merge rate 0.8 | PTv3 · NuScenes validation | peakMem (GB): 6.20 -> 0.92 | Helped |
| Add token merging at merge rate 0.8 | PTv3 · NuScenes validation | mIoU: 80.3 -> 78.0 | Hurt |
| Add token merging at merge rate 0.8 | PTv3 · ScanNet Val | mIoU: 77.6 -> 77.0 | Hurt |
| Fine-tune PTv3 with token merging | PTv3 · ScanNet Val | mIoU: 77.6 -> 77.4 | No clear effect |
| Add token merging at merge rate 0.8 | PTv3 · ScanNet200 Val | mIoU: 35.2 -> 34.4 | Hurt |
| Fine-tune PTv3 with token merging | PTv3 · ScanNet200 Val | mIoU: 35.2 -> 35.2 | No clear effect |
| Add token merging at merge rate 0.8 | PTv3-Sonata · S3DIS Area5 | mIoU: 72.2 -> 72.8 | Helped |
| Add token merging at merge rate 0.8 | SpatialLM · SpatialLM dataset | Time (s): 6.009 -> 4.795 | Helped |
| Replace attention with token shuffling and average pooling (ablation) | PTv3 · ScanNet | mIoU: 77.3 -> 76.2 | Hurt |
| Replace attention with value features rather than retaining spatial information sharing (ablation) | PTv3 · ScanNet | mIoU: 74.8 -> 71.3 | Hurt |
| Train the replacement MLP from scratch instead of fine-tuning pretrained value features (ablation) | PTv3 · ScanNet | mIoU: 71.3 -> 42.7 | Hurt |
| Use global energy rather than local energy (ablation) | PTv3 token merging · ScanNet, r=0.8 | mIoU: 74.9 -> 75.8 | Helped |
Surprising findings
- Replacing attention with parameter-free shuffle-and-pool mixing nearly matches attention performance trained from scratch (76.2 vs 77.3 mIoU), suggesting attention may not be strictly necessary.
- Metric-based destination selection from PiToMe degrades performance compared to random selection on 3D point clouds.
Limitations
- The merging rate r is manually specified rather than learned, and automatic optimization would need differentiable approximations.
- No formal theoretical framework quantifies token redundancy; the transfer entropy analysis is only preliminary.
- Some reported ablations and numbers are inconsistent across sections (e.g., PTv3 memory and FLOPs figures differ between intro and tables).
Experimental setup
Datasets: ScanNet ScanNet200 S3DIS nuScenes ShapeNet Objaverse Google Scanned Objects Waymo
Benchmarks: ScanNet ScanNet200 S3DIS nuScenes Objaverse-OOD GSO-OOD RealWorld-OOD SpatialLM detection F1
Metrics: mIoU mAcc allAcc GFLOPs peak memory latency PSNR SSIM LPIPS F1
Baselines: ToMe PiToMe ALGM Random Drop Farthest Point Sampling VoxelGrid Downsampling MipNeRF360 3DGS 2DGS Nerfbusters LaRa MinkUNet
Builds on: Token Merging (ToMe) PiToMe ALGM Point Transformer v3
Closest papers by meaning
- Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers (NeurIPS 2026 · similarity 0.86)
- Point Transformer V3: Simpler Faster Stronger (CVPR 2024 · similarity 0.85)
- SAM-Guided Masked Token Prediction for 3D Scene Understanding (NeurIPS 2024 · similarity 0.85)
- OneFormer3D: One Transformer for Unified Point Cloud Segmentation (CVPR 2024 · similarity 0.85)
- Accelerating Transformers with Spectrum-Preserving Token Merging (NeurIPS 2024 · similarity 0.84)
- TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.