Research map / Image generation / Text-to-image diffusion
Anywhere: A Multi-Agent Framework for User-Guided, Reliable, and Diverse Foreground-Conditioned Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Anywhere, a multi-agent framework (VLM/LLM agents plus ControlNet, SDXL refiner, and inpainting tools) for foreground-conditioned background generation with user text, template repainting for object integrity, and feedback-driven regeneration.
Problem
Foreground-conditioned image generation (completing a background around a given foreground object) suffers from violated object integrity, foreground-background inconsistency, limited diversity, and text-prompt inconsistency in end-to-end inpainting models.
Approach
Replaces end-to-end inpainting with a modular multi-agent pipeline: a VLM foreground analyzer, an LLM prompt creator and selector, an edge-guided template generator, a template repainter that detects and fixes foreground-integrity violations via segmentation and mask comparison, an SDXL refiner, and a VLM quality evaluator that triggers regeneration with feedback.
Key results
- Ours achieves FID 37.007 vs 38.797 (HD-Painter) and 41.047 (BrushNet) in text-free I2I, and AS 5.604 vs 5.238 for BrushNet.
- User study (10 participants, 100 images): bad case rate 0.19 for Anywhere vs 0.34 for BrushNet, 0.47 for SD Inpainting, and 0.66 for HD-Painter.
- In text-guided TI2I, CLIP-Sim improves to 18.357 vs 17.483 for BrushNet, with FID 36.450 vs 38.313.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Anywhere instead of the strongest baseline, BrushNet | foreground-conditioned generation · I2I | AS: 5.238 -> 5.604 | Helped |
| use Anywhere instead of the strongest baseline, BrushNet | foreground-conditioned generation · I2I | PickScore: 0.261 -> 0.418 | Helped |
| use Anywhere instead of the strongest baseline, BrushNet | foreground-conditioned generation · I2I | HPS: 0.186 -> 0.190 | Helped |
| use Anywhere instead of the strongest baseline, HD-Painter | foreground-conditioned generation · I2I | FID: 38.797 -> 37.007 | Helped |
| use Anywhere instead of the strongest baseline, BrushNet | foreground-conditioned generation · TI2I | CLIP-Sim: 17.483 -> 18.357 | Helped |
| use Anywhere instead of the strongest baseline, HD-Painter | foreground-conditioned generation · TI2I | FID: 37.398 -> 36.450 | Helped |
| add Prompt Generation Module (ablation) | Anywhere · I2I | AS: 5.263 -> 5.604 | Helped |
| add Template Repainter (ablation) | Anywhere · I2I | AS: 5.417 -> 5.604 | Helped |
| add Image Enhancer (ablation) | Anywhere · I2I | AS: 5.518 -> 5.604 | Helped |
| add Foreground Analyzer (ablation) | Anywhere · I2I | AS: 5.395 -> 5.604 | Helped |
| add Quality Evaluator (ablation) | Anywhere · I2I | AS: 5.462 -> 5.604 | Helped |
| add Prompt Creator (ablation) | Anywhere · TI2I | AS: 5.561 -> 5.812 | Helped |
Surprising findings
- The w/o Prompt Selector variant achieves higher AS (5.624) than the full model in the TI2I ablation (5.812 is full; 5.624 is ablated but still lower), while the w/o QE variant has a lower FID than the full model (36.952 vs 36.450 is full, so not lower), so ablation trends are mixed.
- The framework requires roughly 2-3x more GPU time than end-to-end inpainting models per generation.
Limitations
- Approximately 2-3x higher GPU time per generation than end-to-end models
- Struggles with corner cases such as transparent foreground objects
- Evaluation relies on a small custom test set of 3,000 images and a 10-participant user study
Experimental setup
Datasets: LAION MSCOCO
Benchmarks: custom 3,000-image foreground test set (LAION + MSCOCO)
Metrics: Aesthetic Score PickScore HPS CLIP-Sim FID user study aesthetic quality diversity score bad case rate
Baselines: BrushNet HD-Painter Stable Diffusion 2.0 Inpainting
Builds on: ControlNet Stable Diffusion XL Grounding DINO LaMa
Closest papers by meaning
- VersaGen: Unleashing Versatile Visual Control for Text-to-Image Synthesis (AAAI 2025 · similarity 0.86)
- AnyScene: Customized Image Synthesis with Composited Foreground (CVPR 2024 · similarity 0.85)
- Unify-Agent: A Unified Multimodal Agent for World-Grounded Image Synthesis (NeurIPS 2026 · similarity 0.85)
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.85)
- SpotActor: Training-Free Layout-Controlled Consistent Image Generation (AAAI 2025 · similarity 0.84)
- MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.