Research map / Image generation / Text-to-image diffusion
FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreeScene, which uses a GPT-4o-based Graph Designer to turn free-form text/image inputs into partial scene graphs, and MG-DiT, a mixed discrete-continuous graph diffusion transformer, to generate 3D indoor scenes with constrained sampling for multiple zero-shot applications.
Problem
Indoor 3D scene synthesis offers either rough language control lacking fine-grained customization or graph-based control that requires cumbersome manual graph design; users need convenient yet precise controllability.
Approach
A VLM (GPT-4o) with one-shot chain-of-thought prompting extracts a partial scene graph from free-form text and/or images, and a mixed graph diffusion transformer (MG-DiT) jointly denoises discrete graph/category/feature variables and continuous bounding boxes, keeping the extracted graph fixed through constrained sampling so one model handles text-to-scene, graph-to-scene, re-arrangement, completi
Key results
- Text-to-scene on bedrooms: Ours+GD reaches FID 108 and iRecall 81.40% vs InstructScene FID 114.86 and iRecall 72.71%.
- Graph Designer with CoT raises image relationship accuracy from 34.65% to 77.56% over one-shot prompting without CoT.
- Re-arrangement on bedrooms: FID 105.71 and iRecall 77.49% vs InstructScene 106.80 and 76.24%; comparable stylization results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FreeScene instead of InstructScene for text-to-scene generation | Text-to-scene generation · Bedroom | FID: 114.86 -> 111.21 | Helped |
| Use FreeScene instead of InstructScene for text-to-scene generation | Text-to-scene generation · Livingroom | FID: 111.52 -> 110.55 | Helped |
| Use FreeScene instead of InstructScene for text-to-scene generation | Text-to-scene generation · Diningroom | FID: 129.13 -> 127.28 | Helped |
| Preprocess text with Graph Designer | FreeScene text-to-scene generation · Bedroom | iRecall (%): 73.69 -> 81.40 | Helped |
| Preprocess text with Graph Designer | FreeScene text-to-scene generation · Livingroom | iRecall (%): 58.16 -> 71.81 | Helped |
| Preprocess text with Graph Designer | FreeScene text-to-scene generation · Diningroom | iRecall (%): 63.39 -> 75.01 | Helped |
| Use FreeScene instead of InstructScene for graph-to-scene generation | Graph-to-scene generation · Bedroom | FID: 101.86 -> 98.31 | Helped |
| Use FreeScene instead of InstructScene for graph-to-scene generation | Graph-to-scene generation · Livingroom | FID: 108.45 -> 107.62 | Helped |
| Use FreeScene instead of InstructScene for graph-to-scene generation | Graph-to-scene generation · Diningroom | FID: 123.82 -> 122.57 | Helped |
Surprising findings
- Adding the graph designer to text-to-scene improves controllability but on stylization MG-DiT only matches InstructScene rather than clearly beating it.
Limitations
- Graph Designer is limited by VLM capabilities, especially for scenes with many objects.
- MG-DiT cannot precisely control exact object position and orientation, which can yield impractical arrangements.
- Indoor scene datasets are small and of limited quality, risking overfitting.
Experimental setup
Datasets: 3D-FRONT 3D-FUTURE
Benchmarks: 3D-FRONT
Metrics: FID FIDCLIP KID SCA iRecall Object iRecall Rel Acc CLIP style delta
Baselines: ATISS DiffuScene InstructScene
Builds on: MiDiffusion D3PM InstructScene DiT FiLM
Closest papers by meaning
- InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.90)
- DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis (CVPR 2024 · similarity 0.90)
- MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generation (AAAI 2025 · similarity 0.89)
- R3CD: Scene Graph to Image Generation with Relation-Aware Compositional Contrastive Control Diffusion (AAAI 2024 · similarity 0.88)
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.88)
- SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.