Research map / Vision-language and multimodal / Multimodal LLMs

Part-X-MLLM: Part-aware 3D Multimodal Large Language Model

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Part-X-MLLM, a native 3D part-aware MLLM that emits a single token program of part bounding boxes, text, and edit commands, driving external geometry engines for part-aware generation, grounded QA, and localized editing.

Problem

Existing 3D MLLMs treat objects as monolithic and lack persistent part identifiers, grounded references, and executable outputs for part-level generation and editing.

Approach

Formulate diverse 3D tasks (box listing, grounded QA, generation, add/delete/modify editing) as programs in a structured grammar of quantized BBox tokens and edit operators, generated autoregressively from a dual-encoder (geometry plus RGB) LLM, with the output decoupled from the geometry engines that execute it.

Key results

• Bounding box generation: Part-X-MLLM reaches 42.55% BBox IoU and 74.11% voxel recall vs 37.33% and 69.65% for PartField.
• Part QA on UniPart-Bench: SBERT 78.98 and SimCSE 84.25, versus best baseline ShapeLLM-13B at 61.19 SBERT.
• Dual-encoder ablation: +7.06 IoU on pure box listing over a single-encoder variant (75.53 vs 68.47).

Limitations

• Longer token sequences slow inference.
• Confidence-based face segmentation from boxes is relatively shallow.
• Fine-tuning on 3D tasks may reduce the base LLM's general language capabilities.

Experimental setup

Datasets: in-house part-centric dataset (85,771 objects)

Benchmarks: UniPart-Bench

Metrics: IoU voxel recall voxel IoU SBERT SimCSE BLEU-1 ROUGE-L METEOR

Baselines: PartField OmniPart (SAM Mask) GPT4Point PointLLM-7B PointLLM-13B ShapeLLM-13B ShapeLLM-Omni-7B

Builds on: Qwen2.5-VL TRELLIS OmniPart

Closest papers by meaning

• 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models (NeurIPS 2026 · similarity 0.92)
• CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models (ICML 2026 · similarity 0.90)
• ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding (NeurIPS 2025 · similarity 0.89)
• Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.89)
• MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation (NeurIPS 2025 · similarity 0.88)
• Merge3D: Efficient 3D Multimodal LLMs via Joint 2D-3D Token Merging (CVPR 2026 · similarity 0.88)