Research map / Point clouds / Point cloud classification
PerLA: Perceptive 3D Language Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PerLA, a 3D language assistant that splits a point cloud into Hilbert-curve-ordered parts, encodes local and global views, and fuses them via localized cross-attention and GCN message passing with a local consensus loss.
Problem
Current 3D language assistants downsample point clouds or split them into parts, losing fine-grained local details or global context needed for accurate 3D question answering and dense captioning.
Approach
A perceptive scene encoder that encodes equal-cardinality Hilbert-serialized local partitions in parallel with the downsampled global scene, links them by Hilbert-based k-NN, aggregates them with localized cross-attention and GCN message passing, and regularizes with a local consensus loss, adding no extra LLM tokens.
Key results
- On ScanQA test w/ object, PerLA reaches 80.91 CiDEr, +1.34 over LL3DA on validation (78.13 vs 76.79).
- On ScanRefer@0.5, PerLA reaches 69.41 CiDEr, +4.22 over LL3DA (65.19).
- On Nr3D@0.5, PerLA reaches 55.06 CiDEr, +3.88 over LL3DA (51.18).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine local and global representations instead of using either representation alone (ablation) | PerLA · ScanQA, ScanRefer@0.5, Nr3D@0.5 | CIDEr | Helped |
| Add GCN message passing to refine global representations (ablation) | PerLA · ScanQA, ScanRefer@0.5, Nr3D@0.5 | CIDEr: 77.61 -> 78.13; 69.04 -> 69.41; 54.30 -> 55.06 | Helped |
| Replace localized cross-attention with mean pooling (ablation) | PerLA · ScanQA validation | CIDEr: 74.4 vs 78.1 | Hurt |
| Replace localized cross-attention with max pooling (ablation) | PerLA · ScanQA validation | CIDEr: 74.0 vs 78.1 | Hurt |
| Add the smoothness loss to the prediction loss (ablation) | PerLA · ScanQA validation | CIDEr: 75.31 -> 76.62 | Helped |
| Add the regularization loss alongside smoothness and prediction losses (ablation) | PerLA · ScanQA validation | CIDEr: 76.62 -> 78.13 | Helped |
| Use 6 rather than 4 or 8 point-cloud partitions (ablation) | PerLA · ScanQA validation | CIDEr: 76.9; 78.1; 78.0 | Mixed |
| Increase query tokens using local partitions and whole-scene tokens instead of PerLA's local-to-global aggregation (ablation) | LL3DA† · ScanQA, ScanRefer@0.5, Nr3D@0.5 | CIDEr: 74.54 vs 78.13; 62.25 vs 69.41; 48.70 vs 55.06 | Hurt |
Surprising findings
- Simply increasing the number of query tokens for local partitions (LL3DA†) does not match PerLA, indicating that information exchange between local and global representations matters more than token count.
Limitations
- Evaluated only on ScanNet-based benchmarks (ScanQA, ScanRefer, Nr3D); robustness in more complex 3D scenarios is untested.
- Relies on a frozen OPT-1.3B LLM and a pre-trained 3D scene encoder; the gains are reported in relative terms with a reproduced LL3DA baseline.
Experimental setup
Datasets: ScanNet ScanQA ScanRefer Nr3D 3D-LLM
Benchmarks: ScanQA ScanRefer Nr3D
Metrics: CiDEr BLEU-4 METEOR ROUGE-L m@kIoU
Baselines: ScanQA 3D-LLM LL3DA 3D-VLP 3D-VisTA Scan2Cap Vote2CapDETR 3DJCG UniT3D LL3DA†
Builds on: LL3DA BLIP-2 3D-LLM Octree-based Hilbert serialization (Point Transformer V3)
Closest papers by meaning
- More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding (AAAI 2025 · similarity 0.90)
- 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models (NeurIPS 2026 · similarity 0.89)
- LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding Reasoning and Planning (CVPR 2024 · similarity 0.88)
- Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers (NeurIPS 2024 · similarity 0.88)
- Part-X-MLLM: Part-aware 3D Multimodal Large Language Model (ICLR 2026 · similarity 0.88)
- LangSplat: 3D Language Gaussian Splatting (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.