Research map / Image generation / Editing and style transfer
SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SceneDesigner, a ControlNet-style branch on Stable Diffusion 3.5 that conditions on a cuboid NOCS map to control 9-DoF poses of multiple objects, trained with a new ObjectPose9D dataset plus an RL fine-tuning stage and Disentangled Object Sampling at inference.
Problem
Simultaneous control of location, size, and orientation (9D poses) of multiple objects in text-to-image generation, where existing methods lack controllability and image quality.
Approach
Encode 9D object poses as a camera-view Cuboid NOCS (CNOCS) map fed to a ControlNet-like branch, fine-tune with a reward-based RL stage on rebalanced pose data, and at inference combine per-object and global noisy latents using region masks (Disentangled Object Sampling).
Key results
- On ObjectPose-Single-Front, SceneDesigner reaches 50.20% Acc_ls and 13.23 Abs.Err versus 23.89% and 87.26 for LOOSECONTROL.
- FID improves to 24.91 and CLIP to 0.345, compared with 37.89 and 0.293 for LOOSECONTROL.
- On ObjectPose-Multi, Acc_ls is 47.16% and Acc@22.5° is 80.79%, versus 14.85% and 4.80% for LOOSECONTROL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MS-COCO training data (ablation) | SceneDesigner · ObjectPose-Single | Acc_ls (%): 41.69 -> 51.12 | Helped |
| add RL finetuning (ablation) | SceneDesigner · ObjectPose-Single | Acc_ls (%): 43.18 -> 51.12 | Helped |
| use I-CNOCS instead of C-CNOCS (ablation) | SceneDesigner · ObjectPose-Single | Acc_ls (%): 40.45 -> 51.12 | Helped |
| use I-CNOCS instead of pose embedding (ablation) | SceneDesigner · ObjectPose-Single | Acc_ls (%): 32.51 -> 51.12 | Helped |
| use I-CNOCS instead of prompt-only pose description (ablation) | SceneDesigner · ObjectPose-Single | Acc_ls (%): 12.90 -> 51.12 | Helped |
| add Disentangled Object Sampling (ablation) | SceneDesigner · ObjectPose-Multi | Acc_ls (%): 36.68 -> 47.16 | Helped |
| add L_prior (ablation) | SceneDesigner · ObjectPose-Single | Acc_ls (%): 26.55 -> 51.12 | Helped |
Limitations
- Cannot control the precise shape of objects.
- Multi-object performance is bounded by the base model; attribute leakage persists with many concepts.
- Disentangled Object Sampling introduces extra computation.
Experimental setup
Datasets: OmniNOCS MS-COCO ObjectPose9D
Benchmarks: ObjectPose-Single-Front ObjectPose-Single-Back ObjectPose-Multi
Metrics: Acc_ls mIoU Abs.Err Acc@22.5° FID CLIP score
Baselines: Continuous 3D Words LOOSECONTROL
Builds on: ControlNet NOCS AlignProp LOOSECONTROL
Closest papers by meaning
- Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models (NeurIPS 2024 · similarity 0.86)
- From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries (NeurIPS 2025 · similarity 0.85)
- DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Data (NeurIPS 2025 · similarity 0.85)
- ObjectMover: Generative Object Movement with Video Prior (CVPR 2025 · similarity 0.85)
- RecDreamer: Consistent Text-to-3D Generation via Uniform Score Distillation (ICLR 2025 · similarity 0.85)
- Disentangled Pose and Appearance Guidance for Multi-Pose Generation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.