Research map / Image generation / Text-to-image diffusion
AnyScene: Customized Image Synthesis with Composited Foreground
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AnyScene, a ControlNet-based foreground injection framework with cross-attention layout control and Poisson-style foreground blending to generate diverse scenes around user-composited foregrounds, evaluated on a new 3,108-pair COCO-based benchmark.
Problem
Generating varied, harmonious scenes around user-specified composited foreground elements from text prompts, while avoiding foreground distortion, redundant regeneration of foreground elements, and loss of fine foreground detail.
Approach
A foreground injection module (ControlNet-style encoder trained with denoising, masked content, and edge-gradient losses) conditions a frozen Stable Diffusion; at inference, cross-attention maps of foreground words are masked to the foreground region, and the generated scene is blended with the original foreground details.
Key results
- On the proposed benchmark, AnyScene achieves FID 16.09 and CLIP-S 15.18, versus 16.52 / 14.72 for Inpaint Anything and 24.07 / 14.64 for Canny Edge ControlNet.
- AnyScene's aesthetic score is 5.94, close to default Stable Diffusion (6.02) and above Inpaint Anything (5.83) and Canny Edge (5.56).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AnyScene instead of Inpainting Anything | Stable Diffusion v1.5 · AnyScene benchmark | FID: 16.52 -> 16.09 | Helped |
| Use AnyScene instead of Inpainting Anything | Stable Diffusion v1.5 · AnyScene benchmark | CLIP-S: 14.72 -> 15.18 | Helped |
| Use AnyScene instead of Stable Diffusion | Stable Diffusion v1.5 · AnyScene benchmark | Aesthetic: 6.02 -> 5.94 | Hurt |
| Add content loss to the denoising loss (ablation) | AnyScene Foreground Injection Module · Customized scene synthesis | foreground texture and color fidelity | Helped |
| Add edge gradient loss to the denoising loss (ablation) | AnyScene Foreground Injection Module · Customized scene synthesis | foreground edge realism | Helped |
| Apply foreground layout control to confine foreground semantics to the mask (ablation) | AnyScene · Customized scene synthesis | foreground distortion and redundant generation | Helped |
| Use foreground blending instead of direct copy and paste (ablation) | AnyScene · Customized scene synthesis | visual harmony | Helped |
Limitations
- Limited variability in the number and type of foreground elements, per the authors.
- Evaluation relies on generic metrics (FID, CLIP-Score, aesthetic) with no human study or foreground-fidelity metric.
- Subject-driven baselines are compared qualitatively only, not quantitatively.
Experimental setup
Datasets: FFHQ COCO OpenImages
Benchmarks: AnyScene benchmark (COCO-based, 3,108 pairs)
Metrics: FID CLIP-Score LAION Aesthetic
Baselines: Stable Diffusion Canny Edge ControlNet Inpaint Anything BLIP-Diffusion DreamBooth Custom Diffusion
Builds on: ControlNet Stable Diffusion Poisson image blending Prompt-to-Prompt attention control
Closest papers by meaning
- FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts (CVPR 2025 · similarity 0.86)
- Anywhere: A Multi-Agent Framework for User-Guided, Reliable, and Diverse Foreground-Conditioned Image Generation (AAAI 2025 · similarity 0.85)
- Generating compositional scenes via Text-to-image RGBA Instance Generation (NeurIPS 2024 · similarity 0.84)
- TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generation (ICML 2025 · similarity 0.84)
- LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integration (NeurIPS 2025 · similarity 0.83)
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.