Research map / Vision-language and multimodal / Multimodal LLMs

3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Makes parts addressable in a 3D MLLM by reserving per-region vocabulary tokens over an unsupervised geometric partition, refined by a lightweight Marker-Space Refinement module, without a segmentation decoder or box head.

Problem

Object-level 3D MLLMs treat an object as a flat patch-token sequence and cannot address, name, or reason about its parts; prior part-aware methods add segmentation decoders or box grammars at large parameter cost.

Approach

Partition the frozen point encoder's patch tokens into K=16 deterministic geometric regions, insert per-region learnable markers and reserved vocabulary tokens <part_k> before each region's patches, and refine markers with MSR (conditioning on region statistics and adjacency), so parts become readable and writable LLM tokens.

Key results

• On PartVerse-QA caption-to-slots, 3D-PLOT-LLM reaches Jaccard 0.459 and Exact-match 13.78%, a +64% relative gain over the vocab-only variant.
• On 3DCoMPaT-GrIn PaPGD, 3D-PLOT-LLM outperforms PointLLM on all five reported metrics, including +3.03 GPT-4o judge (43.34 vs 40.31).
• On Objaverse captioning, the full model with PartVerse-QA reaches +0.65 SBERT and +1.85 GPT-4o over PointLLM, adding under 1M trainable parameters.

Surprising findings

• Raw markers without MSR regress below the vocab-only variant on PartVerse-QA (Jaccard 0.263 vs 0.413), indicating unconditioned markers distract the model.
• A label-permutation probe drops Jaccard from 0.459 to 0.209, suggesting the learned token-region binding is real rather than cosmetic.

Limitations

• Parts are addressed at K=16 slot granularity rather than point-wise masks, so it is not a substitute for segmentation-decoder methods.
• Direct cross-method comparison on PartVerse-QA is structurally constrained; prior methods are not evaluated on it.
• Fixed slot budget K=16 covers 94.5% of objects; variable-K not explored.

Experimental setup

Datasets: Cap3D Objaverse PointLLM complex instructions PartVerse 3DCoMPaT-GrIn

Benchmarks: PartVerse-QA 3DCoMPaT-GrIn PaPGD Objaverse captioning

Metrics: Jaccard Exact-match SBERT SimCSE GPT-4o judge BLEU-1 ROUGE-L METEOR BLEU-4 Word-F1

Baselines: PointLLM ShapeLLM PointLLM-PiSA Kestrel PARIS3D SegPoint

Builds on: PointLLM Point-BERT Vicuna Cap3D PartVerse

Closest papers by meaning

• Part-X-MLLM: Part-aware 3D Multimodal Large Language Model (ICLR 2026 · similarity 0.92)
• More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding (AAAI 2025 · similarity 0.92)
• Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.90)
• Merge3D: Efficient 3D Multimodal LLMs via Joint 2D-3D Token Merging (CVPR 2026 · similarity 0.90)
• Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.90)
• CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models (ICML 2026 · similarity 0.90)