Research map / Multi-agent systems and games / LLM agents
ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ShapeCraft, a multi-agent LLM framework (Parser, Coder, Evaluator) that builds a graph-based procedural shape representation in Blender for structured, textured, interactive text-to-3D generation.
Problem
Text-to-3D methods yield unstructured, poorly editable meshes, and LLMs struggle to translate long, complex text with spatial relations into accurate procedural shape programs.
Approach
A flat depth-1 Graph-based Procedural Shape (GPS) representation with per-component descriptions, bounding volumes and Blender code, built hierarchically by a Parser agent, refined by a Coder agent with multi-path sampling and visual feedback from an Evaluator, plus component-aware SDS for BRDF texturing.
Key results
- ShapeCraft achieves IoGT 0.471 and CLIP score 27.27, best among compared methods, with VQA pass rate 0.44 versus 0.42 for MVDream.
- MVDream attains the best Hausdorff distance (0.411) vs ShapeCraft's 0.415, while ShapeCraft runs in 11.68 min versus 32.10 min for MVDream.
- Using the GPS parser, ShapeCraft reaches a 100% compile rate, while thinking-mode LLMs (o3, o4-mini-high, DeepSeek-R1, Gemini-2.5-Pro) range from 60% to 80%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ShapeCraft instead of BlenderLLM | MARVEL subset · MARVEL subset | IoGT: 0.455 -> 0.471 | Helped |
| Use ShapeCraft instead of MVDream | MARVEL subset · MARVEL subset | Hausdorff distance: 0.411 -> 0.415 | Hurt |
| Use ShapeCraft instead of BlenderLLM | MARVEL subset · MARVEL subset | CLIP Score: 26.99 -> 27.27 | Helped |
| Use ShapeCraft instead of BlenderLLM | MARVEL subset · MARVEL subset | VQA Pass Rate: 0.43 -> 0.44 | Helped |
| Use ShapeCraft instead of 3D-PREMISE | MARVEL subset · MARVEL subset | Run Time: 2.81 min. -> 11.68 min. | Hurt |
| Use ShapeCraft instead of 3D-PREMISE | MARVEL subset · MARVEL subset | API Calls: 6 -> 21 | Hurt |
| Increase sampled paths from M=1 to M=3 at T=1 (ablation) | ShapeCraft · Shape modeling ablation | IoGT: 0.436 -> 0.535 | Helped |
| Increase sampled paths from M=1 to M=3 at T=1 (ablation) | ShapeCraft · Shape modeling ablation | Hausdorff: 0.485 -> 0.444 | Helped |
| Increase sampled paths from M=1 to M=3 at T=1 (ablation) | ShapeCraft · Shape modeling ablation | CLIP Score: 25.75 -> 25.90 | Helped |
| Increase iterative updates from T=1 to T=3 at M=1 (ablation) | ShapeCraft · Shape modeling ablation | Hausdorff: 0.485 -> 0.494 | Hurt |
| Increase iterative updates from T=1 to T=3 at M=1 (ablation) | ShapeCraft · Shape modeling ablation | IoGT: 0.436 -> 0.492 | Helped |
| Increase iterative updates from T=1 to T=3 at M=1 (ablation) | ShapeCraft · Shape modeling ablation | CLIP Score: 25.75 -> 26.20 | Helped |
Surprising findings
- Free-form thinking-mode LLMs such as Gemini-2.5-Pro and o3 score worse than the GPS-based pipeline on IoGT, indicating explicit hierarchical parsing constrains reasoning better than CoT.
- Deepseek-R1-0528 achieves the highest CLIP score (29.01) among all methods, above ShapeCraft's 27.27.
Limitations
- Sensitive to ambiguous, brief, or creative prompts that degrade parsing and visual feedback.
- Struggles with complex or organic geometry (tails, wings) due to limited Coder library scope.
- Evaluated on only 26 prompts; reproduced baselines (3D-PreMise, CADCodeVerify) since originals were not open-source.
Experimental setup
Datasets: MARVEL-40M+
Benchmarks: MARVEL subset (26 prompts)
Metrics: IoGT Hausdorff distance CLIP score VQA pass rate run time API calls compile rate
Baselines: MVDream LLaMA-Mesh 3D-PreMise CADCodeVerify L3GO BlenderLLM
Builds on: 3D-PREMISE CADCodeVerify SDS xatlas
Closest papers by meaning
- SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code (ICML 2024 · similarity 0.89)
- 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code (NeurIPS 2026 · similarity 0.89)
- ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding (NeurIPS 2025 · similarity 0.87)
- Muses: 3D-Controllable Image Generation via Multi-Modal Agent Collaboration (AAAI 2025 · similarity 0.87)
- VLMaterial: Procedural Material Generation with Large Vision-Language Models (ICLR 2025 · similarity 0.86)
- ShapeWalk: Compositional Shape Editing Through Language-Guided Chains (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.