Research map / Image generation / Text-to-image diffusion
Compass Control: Multi Object Orientation Control for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compass Control conditions Stable Diffusion on per-object orientation via learned compass tokens from a light MLP encoder, with coupled cross-attention localization (CALL), trained on a small synthetic Blender+ControlNet dataset.
Problem
Text-to-image diffusion models cannot precisely control the 3D orientation of each object in multi-object scenes; text prompts like "facing right" are imprecise and existing 3D-control methods need multi-view images, 3D boxes, or handle only single objects.
Approach
Encode each object's orientation angle as a compass token inserted before its object token in the prompt, and constrain the cross-attention of the compass and object tokens to a loose 2D bounding box (CALL) during training and inference, which localizes orientation influence and disentangles objects; combined with staged training (single-object then mixed scenes).
Key results
- Single-object: Angular error 0.198 for Ours vs 0.385 for LooseControl, 0.509 for Cont-3D-Words, 0.596 for ViewNeTI.
- Multi-object: Angular error 0.215 for Ours vs 0.372 for LooseControl; % object generated 0.964 vs 0.778.
- User study with 57 participants preferred Compass Control over baselines across text alignment, object quality, and orientation consistency (e.g., 86-98% preference in several comparisons).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Compass Control instead of ViewNeTI | Single-object generation · Single object | Angular Err.: 0.596 -> 0.198 | Helped |
| Use Compass Control instead of Cont-3D-words | Single-object generation · Single object | % Obj. Generated: 0.732 -> 0.968 | Helped |
| Use Compass Control instead of LooseControl | Single-object generation · Single object | Angular Err.: 0.385 -> 0.198 | Helped |
| Use Compass Control instead of LooseControl | Multi-object generation · Multiple object | % Obj. Generated: 0.778 -> 0.964 | Helped |
| Use Compass Control instead of LooseControl | Multi-object generation · Multiple object | Angular Err.: 0.372 -> 0.215 | Helped |
| Add staged training (ablation) | Compass Control · Scenes with one, three, and five objects | Bounding-box adherence | Helped |
| Add CALL attention localization (ablation) | Compass Control · Multi-object scenes | Orientation control | Helped |
| Add ControlNet augmentations (ablation) | Compass Control · Scenes with diverse contexts | Background diversity | Helped |
Surprising findings
- Without CALL, compass tokens attend to irrelevant regions and objects entangle, reducing the number of objects generated.
- Single-stage training leads to objects leaking outside bounding boxes and suppressing neighboring objects.
Limitations
- Struggles with occluded or heavily overlapping objects, leading to missing objects or attribute mixing.
- Single-angle orientation representation is too simplistic for non-rigid objects such as humans.
- Training limited to 10 synthetic 3D assets and one- or two-object scenes.
Experimental setup
Datasets: Synthetic Blender-rendered scenes of 10 3D assets ControlNet Canny-augmented scenes
Benchmarks: Custom test set of 10 ChatGPT-generated scene prompts with seen and unseen objects
Metrics: CLIP text alignment % object generation (Grounding-DINO) angular error user study preference
Baselines: Cont-3D-Words ViewNeTI LooseControl CD-360
Builds on: Stable Diffusion LoRA ControlNet DreamBooth Cont-3D-Words
Closest papers by meaning
- Generating compositional scenes via Text-to-image RGBA Instance Generation (NeurIPS 2024 · similarity 0.87)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.87)
- Generating Images with 3D Annotations Using Diffusion Models (ICLR 2024 · similarity 0.86)
- Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models (NeurIPS 2024 · similarity 0.86)
- Local Conditional Controlling for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.86)
- Directed Diffusion: Direct Control of Object Placement through Attention Guidance (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.