Research map / 3D vision / 3D generation
FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FirePlace places new 3D objects into preexisting scenes by having an off-the-shelf MLLM (Gemini-1.5 Pro) generate constraint outlines, resolve them to 3D surfaces via batched visual selection, solve geometric constraints, and prune placements for common sense.
Problem
Object placement in complex preexisting 3D scenes requires both common-sense semantic reasoning and fine-grained 3D geometric reasoning, which MLLMs lack when working from bounding boxes or raw prompts.
Approach
Use an off-the-shelf MLLM as a planner that emits geometric constraint functions over fine-grained interaction surfaces extracted from meshes, selects instances and surfaces via visual selection in batches (inference-compute scaling), then solves constraints and uses MLLM-based plausibility pruning, all training-free.
Key results
- FirePlace reduces min L2 error to 48.39 cm vs 89.01 cm for LayoutGPT and 113.17 cm for Holodeck.
- FirePlace achieves plausibility score 2.95 vs 2.14 (LayoutGPT) and 2.13 (Holodeck) and energy score 0.42 vs 0.22 for Holodeck.
- In a 30-participant user study (2075 comparisons), FirePlace wins 60-66% vs Holodeck and 76% vs LayoutGPT across physics, semantics and common sense axes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use geometric constraint solving rather than selecting among random placements (ablation) | FirePlace · Placement tasks | Mean L2 error: 124.11 -> 62.52 | Helped |
| Use visual anchor-object selection instead of caption-based selection (ablation) | FirePlace · Placement tasks | Energy score: 0.25 -> 0.34 | Helped |
| Use extracted interaction surfaces instead of bounding-box surfaces (ablation) | FirePlace · Placement tasks | Plausibility score: 2.89 -> 2.94 | Helped |
| Add common-sense plausibility pruning (ablation) | FirePlace · Placement tasks | Plausibility score: 2.60 -> 2.94 | Helped |
| Use small batched visual selection rather than selecting among up to 100 options at once (ablation) | FirePlace · Placement tasks | Energy score: 0.19 -> 0.34 | Helped |
| Use FirePlace instead of Holodeck | 3D object placement · 266 placement tasks | Mean L2 error (cm): 137.60 -> 69.89 | Helped |
| Use FirePlace instead of Holodeck | 3D object placement · 266 placement tasks | Visibility score: 0.63 -> 0.88 | Helped |
| Use FirePlace instead of Holodeck | 3D object placement · 266 placement tasks | Plausibility score: 2.13 -> 2.95 | Helped |
| Use FirePlace instead of Holodeck | 3D object placement · 266 placement tasks | Energy score: 0.22 -> 0.42 | Helped |
Surprising findings
- Randomly sampling 100 placements and letting the MLLM choose yields only 24% visibility and much worse L2 error, showing constraints are needed.
- Scaling batch size up to 100 options for visual selection substantially degrades accuracy, so batched selection is important.
Limitations
- Relies on a small library of planar binary constraints, which may not capture all placement relations.
- Depends on proprietary Gemini-1.5 Pro and multiple MLLM calls per placement.
- Evaluated only on 50 scenes with fixed camera viewpoints.
Experimental setup
Datasets: 50 photorealistic 3D scenes designed by human experts (SpatialVerse assets)
Benchmarks: 266 object placement tasks across 50 3D scenes
Metrics: Min L2 error Mean L2 error Energy score Plausibility score Visibility score user preference win rate
Baselines: Holodeck LayoutGPT
Builds on: Holodeck LayoutGPT simulated annealing layout solving
Closest papers by meaning
- LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models (CVPR 2025 · similarity 0.89)
- MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation (NeurIPS 2025 · similarity 0.88)
- S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance (CVPR 2026 · similarity 0.88)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.88)
- Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.87)
- Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.