Research map / Vision-language and multimodal / Multimodal LLMs
ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds ShapeLLM-Omni, a native multimodal LLM that generates and understands 3D shapes by tokenizing voxels with a 3D VQVAE and fine-tuning Qwen2.5-VL-7B on the 3D-Alpaca instruction dataset.
Problem
Multimodal LLMs handle text and images but cannot natively understand or generate 3D content, and 3D-LLM work lacks discrete 3D tokenization and instruction data covering generation, understanding, and editing.
Approach
Voxelize 3D assets at 64^3, compress them with a 3D VQVAE into 1024 discrete tokens (8192 codebook), and train Qwen2.5-VL-7B with next-token prediction over mixed text/image/3D sequences, using a new 2.56M-sample 3D-Alpaca dataset that includes 70k 3D editing pairs.
Key results
- Text-to-3D: ShapeLLM-Omni reaches CLIP 26.7 and FD 25.9 vs. SAR3D 23.9 / 27.2; TRELLIS is best at 30.8 / 18.3.
- 3D captioning on Objaverse: BLEU-1 18.51 and ROUGE-L 21.37, best among listed models, while PointLLM-13B with short captions scores higher on Sentence-BERT and SimCSE.
- Language retention: ShapeLLM-Omni-7B scores MMLU 63.9 vs. Qwen2.5-VL-7B 66.9, PIQA 78.6 vs. 81.0, and GSM8K 55.1 vs. 42.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use unified ShapeLLM-Omni rather than Trellis for text-to-3D generation | 3D generation · Toys4K test, text-to-3D | CLIP: 30.8 -> 26.7 | Hurt |
| use unified ShapeLLM-Omni rather than Trellis for text-to-3D generation | 3D generation · Toys4K test, text-to-3D | FD_incep: 18.3 -> 25.9 | Hurt |
| use unified ShapeLLM-Omni rather than Trellis for text-to-3D generation | 3D generation · Toys4K test, text-to-3D | KD_incep: 0.19 -> 0.25 | Hurt |
| use unified ShapeLLM-Omni rather than Trellis for image-to-3D generation | 3D generation · Toys4K test, image-to-3D | CLIP: 85.0 -> 84.5 | Hurt |
| use unified ShapeLLM-Omni rather than Trellis for image-to-3D generation | 3D generation · Toys4K test, image-to-3D | FD_incep: 8.31 -> 12.2 | Hurt |
| use unified ShapeLLM-Omni rather than Trellis for image-to-3D generation | 3D generation · Toys4K test, image-to-3D | KD_incep: 0.07 -> 0.09 | Hurt |
| fine-tune Qwen2.5-vl-7B on 3D-Alpaca | Qwen2.5-vl-7B · MMLU | score: 66.9 -> 63.9 | Hurt |
| fine-tune Qwen2.5-vl-7B on 3D-Alpaca | Qwen2.5-vl-7B · PIQA | score: 81.0 -> 78.6 | Hurt |
| fine-tune Qwen2.5-vl-7B on 3D-Alpaca | Qwen2.5-vl-7B · GSM8K | score: 42.9 -> 55.1 | Helped |
| fine-tune Qwen2.5-vl-7B on 3D-Alpaca | Qwen2.5-vl-7B · SIQA | score: 40.7 -> 41.0 | Helped |
| use codebook size 8192 instead of 4096 (ablation) | 3D VQVAE · 3D VQVAE reconstruction, test meshes | Chamfer Distance: 0.0102 -> 0.0094 | Helped |
| use codebook size 8192 instead of 4096 (ablation) | 3D VQVAE · 3D VQVAE reconstruction, test meshes | Hausdorff Distance: 0.0561 -> 0.0525 | Helped |
Surprising findings
- ShapeLLM-Omni's GSM8K score (55.1) exceeds Qwen2.5-VL-7B's (42.9) despite fine-tuning on 3D data, while MMLU and PIQA drop.
Limitations
- Only 70k 3D editing pairs, far too few to reach ChatGPT-4o-level editing.
- Only 7B parameters due to limited compute.
- Generation quality trails TRELLIS, which uses separate models and rectified flow.
Experimental setup
Datasets: 3D-Alpaca Trellis 3D assets (712k) Objaverse-XL UltraChat Toys4K Objaverse
Benchmarks: Toys4K Objaverse captioning (PointLLM benchmark) SIQA PIQA MMLU GSM8K
Metrics: CLIP score FD_incep KD_incep BLEU-1 ROUGE-L METEOR Sentence-BERT similarity SimCSE similarity Chamfer Distance Hausdorff Distance
Baselines: CRM SAR3D 3DTopia-XL TRELLIS PointLLM-13B LLaMA-Mesh Chameleon-7B Qwen2.5-VL-7B
Builds on: Qwen2.5-VL TRELLIS LLaMA-Mesh PointLLM UltraChat
Closest papers by meaning
- CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models (ICML 2026 · similarity 0.90)
- 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models (NeurIPS 2026 · similarity 0.89)
- Part-X-MLLM: Part-aware 3D Multimodal Large Language Model (ICLR 2026 · similarity 0.89)
- SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAE (CVPR 2025 · similarity 0.89)
- ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding (CVPR 2024 · similarity 0.89)
- Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.