Research map / 3D vision / 3D generation
$\textit{Bifr\"ost}$: 3D-Aware Image Compositing with Language Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Bifröst fine-tunes an MLLM (LLaVA) on a counterfactual COCO-based dataset to predict 2.5D object locations (bounding box plus depth) from language instructions, then uses depth maps, DINOv2 ID tokens, and high-frequency detail maps to condition a Stable Diffusion 2.1 compositing model.
Problem
Object-level image compositing is done in 2D and fails on 3D spatial relations such as occlusion, and existing methods lack instruction-based location prediction and identity preservation at once.
Approach
Predict a 2.5D location (bounding box and depth) with a fine-tuned MLLM from language instructions, then inject the resulting depth map as an extra condition alongside ID tokens and high-frequency detail maps in a diffusion compositing model.
Key results
- MLLM location predictor reaches bounding-box IoU 0.8515 versus 0.7567 for LLaVA baseline and 0.0175 for MiniGPTv2.
- On DreamBooth-based compositing, Bifröst reports DINO-score 77.746, CLIP-score 89.722, and FID 15.025, the best of the compared methods including AnyDoor (76.807, 88.735, 15.858).
- User study (30 annotators) gives 3D Awareness 4.21 with depth versus 3.56 for the best baseline Paint-by-Example; without depth, 3D Awareness is 2.56.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune LLaVA to predict 2.5D locations | LLaVA · 2.5D location test set | BBox (MSE): 0.0653 -> 0.0496 | Helped |
| fine-tune LLaVA to predict 2.5D locations | LLaVA · 2.5D location test set | BBox (IoU): 0.7567 -> 0.8515 | Helped |
| add 3D-aware depth-conditioned compositing | Image compositing model · DreamBooth compositing test set | DINO-score: 76.807 -> 77.746 | Helped |
| add 3D-aware depth-conditioned compositing | Image compositing model · DreamBooth compositing test set | CLIP-score: 88.735 -> 89.722 | Helped |
| add video training data (ablation) | Bifröst image compositing model · Image compositing ablation | DINO-score: 68.693 -> 71.573 | Helped |
| add classifier-free guidance (ablation) | Bifröst image compositing model · Image compositing ablation | DINO-score: 71.573 -> 75.578 | Helped |
| add high-frequency filter (ablation) | Bifröst image compositing model · Image compositing ablation | DINO-score: 75.578 -> 76.372 | Helped |
| add depth control (ablation) | Bifröst image compositing model · Image compositing ablation | DINO-score: 76.372 -> 77.746 | Helped |
| add video training data (ablation) | Bifröst image compositing model · Image compositing ablation | CLIP-score: 80.458 -> 83.536 | Helped |
| add classifier-free guidance (ablation) | Bifröst image compositing model · Image compositing ablation | CLIP-score: 83.536 -> 86.394 | Helped |
| add high-frequency filter (ablation) | Bifröst image compositing model · Image compositing ablation | CLIP-score: 86.394 -> 88.634 | Helped |
| add depth control (ablation) | Bifröst image compositing model · User study | 3D Awareness: 2.56 -> 4.21 | Helped |
Surprising findings
- Without depth control, the user-study diversity score (3.61) is higher than with depth (3.07), showing a diversity-spatial control trade-off.
- Paint-by-Example scores higher than the proposed method on user-study diversity (3.87).
Limitations
- MLLM struggles on out-of-distribution objects and more complex scenes, e.g., a backpack placed behind a bed overlapping a chair.
- Depth-map reliance restricts diversity of generated objects in novel poses and views.
- Depth is predicted as a single center-point value per object, not full-object depth.
Experimental setup
Datasets: MS COCO YouTubeVOS MOSE VIPSeg VitonHD MSRA-10K DUT HFlickr LVIS SAM DreamBooth COCO-Val
Benchmarks: DreamBooth (30 concepts x 30 COCO-Val scenes) 855-pair MLLM location test set
Metrics: IoU MSE DINO-score CLIP-score FID user study ratings
Baselines: Paint-by-Example ObjectStitch TF-ICON AnyDoor MiniGPTv2 LLaVA
Builds on: LLaVA Stable Diffusion AnyDoor DINOv2 ControlNet
Closest papers by meaning
- Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents (NeurIPS 2025 · similarity 0.89)
- Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis (ICLR 2024 · similarity 0.88)
- InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.86)
- CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step (NeurIPS 2025 · similarity 0.86)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.86)
- Think, Plan, Paint: Layout-Aware Reasoning for Controllable Image Generation in Unified Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.