Research map / 3D vision / 3D generation
From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FactoredScenes, which generates ScanNet-like 3D rooms by learning a library of layout functions from 3D-Front, using an LLM (GPT-o1) to write room programs regularized by that library, and a program-conditioned attention model to predict object orientations trained on ScanNet.
Problem
Generating realistic real-world (lived-in) indoor scenes with varied object poses is hard because ScanNet-style data is scarce and object orientations are noisy.
Approach
Factor scene generation into learned program library (from synthetic 3D-Front), LLM-generated programs regularized by the library, and a hierarchical pose model that predicts primary object orientations first and then dependent object orientations conditioned on their dependency target's pose, trained on limited ScanNet data.
Key results
- On bedroom layouts vs ScanNet, FactoredScenes reaches FID 67.51 vs 109.40 for the best prior (LayoutGPT), and KID 0.020 vs 0.102.
- The learned library uses 2.89 functions per program on 3D-Front vs 1.07 for an inference-only library, a 170.1% relative improvement.
- In a human study on top-down renders, participants chose the ScanNet room with mean accuracy 0.67, so generated scenes are hard to tell apart from real ones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FactoredScenes instead of LayoutGPT | ScanNet bedroom layouts · Bedroom | FID: 109.40 -> 67.51 | Helped |
| use FactoredScenes instead of LayoutGPT | ScanNet bedroom layouts · Bedroom | KID: 0.102 -> 0.020 | Helped |
| use FactoredScenes instead of Sync2Gen | ScanNet living room layouts · Living room | FID: 139.30 -> 83.49 | Helped |
| use FactoredScenes instead of Sync2Gen | ScanNet living room layouts · Living room | KID: 0.117 -> 0.024 | Helped |
| add predicted object poses rather than use axis-aligned boxes (ablation) | FactoredScenes · Bedroom | FID: 101.55 -> 67.51 | Helped |
| add predicted object poses rather than use axis-aligned boxes (ablation) | FactoredScenes · Bedroom | KID: 0.085 -> 0.020 | Helped |
| replace sampled poses with predicted poses (ablation) | FactoredScenes · Bedroom | FID: 110.03 -> 67.51 | Helped |
| replace sampled poses with predicted poses (ablation) | FactoredScenes · Living room | KID: 0.079 -> 0.024 | Helped |
| add conditioning on dependent targets' predicted poses (ablation) | FactoredScenes pose model · ScanNet test set | Dependent orientation accuracy: 0.397 -> 0.442 | Helped |
| add conditioning on dependent targets' predicted poses (ablation) | FactoredScenes pose model · ScanNet test set | Primary orientation accuracy: 0.537 -> 0.542 | No clear effect |
| learn the program library from example scenes rather than use an inference-only library (ablation) | FactoredScenes program library · 3D-Front | Functions per program: 1.07 -> 2.89 | Helped |
| learn the program library from example scenes rather than use an inference-only library (ablation) | FactoredScenes program library · 3D-Front | mIoU: 0.92 -> 0.96 | Helped |
Surprising findings
- Sampling poses from class-conditioned normal distributions made bedroom FID worse (110.03 vs 101.55 without poses), indicating random perturbations hurt realism.
- The inference-only library achieves a higher mIoU on ScanNet (0.96) than the learned library's lower-compression alternative, because trivial non-compressed solutions reconstruct inputs perfectly.
Limitations
- LLM sometimes generates invalid programs or incorrect dependent-object relations.
- Ground-truth ScanNet orientations come from heuristic tight-box fitting on partial objects, causing inaccurate labels and overlaps.
- No error bars reported because of expensive API-based LLM use.
Experimental setup
Datasets: 3D-Front ScanNet ShapeNet
Benchmarks: ScanNet layouts (FID/KID) ScanNet pose test set
Metrics: FID KID mIoU orientation accuracy NLL human preference accuracy
Baselines: ATISS DiffuScene Sync2Gen LayoutGPT Inference-only library
Builds on: DreamCoder Holodeck LayoutGPT
Closest papers by meaning
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.87)
- Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model (AAAI 2025 · similarity 0.87)
- InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.87)
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.87)
- S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field (AAAI 2025 · similarity 0.87)
- Disentangled 3D Scene Generation with Layout Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.