Research map / Vision-language and multimodal / Human and embodied modeling
Multitwine: Multi-Object Compositing with Text and Layout Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Multitwine is a text- and layout-guided diffusion model that composites multiple objects into a background simultaneously, jointly trained with customization and using attention losses and a VLM-based data pipeline.
Problem
Generative object compositing models insert one object at a time, so they cannot capture interdependent multi-object interactions (e.g., hugging, playing guitar) or add implied props without sequential steps.
Approach
Simultaneous multi-object compositing on Stable Diffusion 1.5 with a multimodal text+image embedding where each object's DINO embedding is inserted after its grounded caption tokens, trained jointly with customization as an auxiliary task and regularized by cross- and self-attention separation losses.
Key results
- On the MultiComp overlapping-box subset, the model reaches DINO 0.532 and CLIP-I 0.741, outperforming single-object baselines such as IMPRINT (0.525 DINO) and AnyDoor (0.520 DINO) run sequentially.
- On MultiBench two-entity customization, the model achieves CLIP-I 0.783 and DINO 0.599 with layout, versus Emu2-Gen's 0.595 and 0.414.
- User studies show preference for the model's realistic interactions up to 97.1% over baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add simultaneous rather than sequential compositing for overlapping objects | Multitwine · MultiComp-overlap | DINO: 0.517 -> 0.532 | Helped |
| Add text guidance for multi-object compositing | Multitwine · MultiComp-overlap | DINO: 0.505 -> 0.532 | Helped |
| Add self-attention loss (ablation) | Multitwine · MultiComp-action | DINO: 0.538 -> 0.540 | Helped |
| Add self-attention loss (ablation) | Multitwine · MultiComp-positional | DINO: 0.584 -> 0.585 | No clear effect |
| Add cross-attention and self-attention losses (ablation) | Multitwine · MultiComp-action | CLIP-Tloc: 0.270 -> 0.286 | Helped |
| Add cross-attention and self-attention losses (ablation) | Multitwine · MultiComp-positional | CLIP-Tloc: 0.281 -> 0.292 | Helped |
| Add joint customization training (ablation) | Multitwine · MultiComp-action | DINO: 0.449 -> 0.540 | Helped |
| Add joint customization training (ablation) | Multitwine · MultiComp-positional | DINO: 0.474 -> 0.585 | Helped |
| Use multi-view training data (ablation) | Multitwine · MultiComp-action | CLIP-Tloc: 0.268 -> 0.286 | Helped |
| Use multi-view training data (ablation) | Multitwine · MultiComp-positional | DINO: 0.591 -> 0.585 | Mixed |
| Use inference attention masking (ablation) | Multitwine · MultiComp-action | DINO: 0.448 -> 0.540 | Helped |
| Use inference attention masking (ablation) | Multitwine · MultiComp-positional | DINO: 0.436 -> 0.585 | Helped |
Surprising findings
- Simultaneous compositing outperforms sequential single-object compositing even with the same model, especially for overlapping boxes.
- Removing customization joint training sharply lowers identity preservation (DINO 0.449 vs 0.540 on MultiComp-action).
Limitations
- Scales poorly to many objects because the multimodal embedding grows with each object.
- Built on SD1.5 due to compute limits, which limits harmonization and composition quality.
- Evaluated mainly on two-object compositing; more objects only emerge qualitatively.
Experimental setup
Datasets: MultiBench MultiComp DreamBooth OpenImages video object relation datasets
Benchmarks: DreamBooth MultiComp MultiBench
Metrics: CLIP-I DINO CLIP-T user preference
Baselines: Paint by Example ControlCom AnyDoor IMPRINT TF-ICON BLIP-Diffusion KOSMOS-G UNIMO-G Emu2-Gen
Builds on: Stable Diffusion ObjectStitch AnyDoor Emu2-Gen
Closest papers by meaning
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.87)
- WeGen: A Unified Model for Interactive Multimodal Generation as We Chat (CVPR 2025 · similarity 0.85)
- DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.85)
- LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integration (NeurIPS 2025 · similarity 0.85)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.85)
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.