Research map / 3D vision / 3D generation
SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SceneCraft, an LLM agent that turns text queries into Blender Python scripts via scene-graph planning, constraint-function coding, visual feedback from GPT-4V, and outer-loop library learning of spatial constraint functions.
Problem
Generating complex 3D scenes with many assets from text requires spatial planning and layout that existing LLM agents (e.g., BlenderGPT) fail to handle, and neural 3D scene generators are limited to narrow domains like indoor scenes.
Approach
A dual-loop pipeline: an inner loop where an LLM planner builds a relational scene graph, an LLM coder translates relations into scoring functions and a constraint-based layout search, and GPT-4V critiques rendered images to revise; an outer loop that merges recurring function edits into a reusable spatial skill library without fine-tuning the LLM.
Key results
- SceneCraft achieves 69.8 CLIP similarity and 88.9 constraint score versus 24.7 and 5.6 for BlenderGPT on held-out synthetic queries.
- In human pairwise evaluation (22 responses), SceneCraft wins 76.8% on text fidelity, 83.6% on composition, and 74.5% on aesthetics against BlenderGPT.
- On Sintel, SceneCraft-guided video generation reaches FVD 317 versus 574 for BlenderGPT and 531 for fine-tuned text-to-video without scene guidance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SceneCraft instead of BlenderGPT | Text-to-scene synthesis · Synthetic queries | Constraint Score: 5.6 -> 88.9 | Helped |
| Use SceneCraft instead of BlenderGPT | Text-to-scene synthesis · Synthetic queries | CLIP SIM: 24.7 -> 69.8 | Helped |
| Use SceneCraft instead of BlenderGPT | Text-to-scene synthesis · Human comparison | Composition win rate: 11.4% -> 83.6% | Helped |
| Use SceneCraft instead of BlenderGPT | Sintel scene planning · Sintel | Scene CLIP SIM: 41.8 -> 82.7 | Helped |
| Use SceneCraft-generated scenes instead of BlenderGPT-generated scenes | VideoPoet fine-tuned on Sintel · Sintel video generation | FVD: 574 -> 317 | Helped |
| Fine-tune VideoPoet on ground-truth scene images | VideoPoet · Sintel video generation | FVD: 846 -> 531 | Helped |
| Add the learned spatial skill library (ablation) | SceneCraft · Synthetic queries | Constraint Score: 64.5 -> 88.9 | Helped |
| Add inner-loop optimization with visual feedback (ablation) | SceneCraft without learned library · Synthetic queries | Constraint Score: 26.1 -> 64.5 | Helped |
| Add the relational scene graph (ablation) | SceneCraft without inner-loop optimization · Synthetic queries | Constraint Score: 3.2 -> 26.1 | Helped |
Surprising findings
- Removing the relation graph drops the constraint score to 3.2, lower than BlenderGPT's 5.6 on the constraint score, suggesting the graph is essential when inner-loop refinement is present.
- On Sintel, BlenderGPT scores higher than SceneCraft on CLIP-based RM (69.1 vs 46.2), despite worse scene layout metrics.
Limitations
- Evaluation uses a small set of 40 manually annotated synthetic queries (20 held out) and only 22 human responses.
- Relies on proprietary GPT-4V and Blender-specific asset pools, limiting reproducibility.
- Sintel experiment assumes ground-truth assets are given, so only layout planning is evaluated.
Experimental setup
Datasets: Sintel movie scenes Turbosquid 3D asset repository synthetic queries with annotated constraints (40 total)
Benchmarks: synthetic queries with annotated constraints (20 held-out) Sintel movie scene-guided video generation
Metrics: CLIP similarity constraint score human win rate layout matrix similarity scene CLIP similarity CLIP-based RM FVD
Baselines: BlenderGPT SceneCraft w/o learned library SceneCraft w/o inner-loop SceneCraft w/o relation graph Text-to-Video (w/o finetune) Text-to-Video (w/ finetune)
Builds on: BlenderGPT WordsEye VideoPoet
Closest papers by meaning
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.90)
- ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling (NeurIPS 2025 · similarity 0.89)
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.88)
- IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering (NeurIPS 2025 · similarity 0.88)
- LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integration (NeurIPS 2025 · similarity 0.87)
- 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.