Research map / Vision-language and multimodal / Multimodal LLMs

CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CG-MLLM, a Qwen3-VL-based multimodal LLM with a Mixture-of-Transformers (TokenAR + BlockAR) and frozen Hunyuan3D-2.1 VAE latents that performs 3D captioning and end-to-end high-resolution 3D generation.

Problem

Current LLM-based 3D generation produces only low-resolution meshes or coarse voxel/lego proxies, and cannot natively generate fine-grained 3D geometry within an MLLM; the paper targets unified 3D captioning and high-fidelity 3D generation.

Approach

Mixture-of-Transformers binds transformers to generation modes rather than tasks: a TokenAR transformer handles serial text/image tokens and a BlockAR transformer handles parallel 3D latent blocks, both initialized from Qwen3-VL with a hybrid causal/parallel attention mask and frozen 3D VAE latents.

Key results

• On image-to-3D generation, CG-MLLM reaches p-FID 12.55 and p-KID 0.27, best among MLLM-based methods (SAR3D 30.07, ShapeLLM-Omni 13.11) but not beating non-MLLM TRELLIS (p-FID 7.36).
• On 3D captioning, Ours-2B MOT obtains BLEU-1 13.51 and ROUGE-L 19.13, beating image-input baselines such as InstructBLIP-13B (4.65) and Qwen3-VL-2B (3.13), but trailing ShapeLLM-Omni-7B (18.51 BLEU-1).
• Ablation: Hunyuan3D-2.1 VAE, MOT, and larger token budget (512 to 4096) each improve p-FID; Qwen3VL-2B at 4096 tokens reaches 12.55 versus 16.57 with Qwen2.5-0.5B.

Surprising findings

• AdaLN conditioning inside the MLLM branches raised training loss substantially above the baseline, which the authors attribute to instability of the shared attention.
• Hunyuan3D-2.1 VAE reconstructions with a mix of uniform and importance points showed holes; the authors used only uniform points.

Limitations

• Still trails leading non-MLLM 3D generators (TRELLIS, Hunyuan3D-2.1) in p-FID and visual quality.
• Limited 3D token budget (fewer than 4k) versus up to ten times more in high-resolution generators; relies on watertight preprocessing that reduces precision.
• Captioning lags specialized 3D captioning models (3D-LLM, PointLLM, ShapeLLM-Omni) due to smaller datasets and model scale; occasional semantic hallucination on ambiguous inputs.

Experimental setup

Datasets: LLaVA-OneVision Trellis-500K Objaverse++ Objaverse-MIX toys4k

Benchmarks: toys4k (Trellis-500K subset)

Metrics: p-FID p-KID CLIP-IQA+ MUSIQ CLIP User Study BLEU-1 ROUGE-L METEOR

Baselines: Michelangelo CraftsMan Hunyuan3D-2.1 TRELLIS SAM3D SAR3D ShapeLLM-Omni LLaVA InstructBLIP Qwen3-VL-2B 3D-LLM PointLLM-13B

Builds on: Mixture-of-Transformers Qwen3-VL Hunyuan3D-2.1 rectified flow

Closest papers by meaning

• ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding (NeurIPS 2025 · similarity 0.90)
• Part-X-MLLM: Part-aware 3D Multimodal Large Language Model (ICLR 2026 · similarity 0.90)
• 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models (NeurIPS 2026 · similarity 0.90)
• Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.90)
• Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.89)
• Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.89)