Research map / Image generation / Text-to-image diffusion
VersaGen: Unleashing Versatile Visual Control for Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VersaGen, a text-to-image agent that lets users give any combination of subject sketches/edge maps and a background drawing, using a trained adaptor on a frozen Stable Diffusion plus three inference-time strategies.
Problem
Existing controllable text-to-image methods require full-scene visual conditions and precise placement, which is inflexible and hard for amateur users; the paper targets versatile control from object-level to scene-level inputs.
Approach
Train a ControlNet-style adaptor on frozen Stable Diffusion with drawings of individual subjects plus background, automatically localise subjects via cross-attention (OTSU thresholding), and apply three inference-time strategies: Multimodal Conflict Resolver (test-time token and pixel losses on latents), Multi-object Decoupling (bounding-box IoU reduction), and Adaptive Control Strength (timestep-
Key results
- On COCO, VersaGen (with LoD from GT) reaches FID 0.3377 and DINO 0.8019 versus ControlNet FID 0.3560 and DINO 0.7774.
- On Sketchy, VersaGen achieves CLIP 0.2524, FID 0.2614 and DINO 0.8193, the best on those metrics, though InstanceDiffusion has higher ACC (0.8942 vs 0.8328).
- In a 50-person user study, 48% of users favored VersaGen as the best balance of ease of use and generation capability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VersaGen with automatically predicted drawing locations instead of ground-truth locations | VersaGen · COCO | FID: 0.3377 -> 0.3431 | Hurt |
| Use VersaGen with automatically predicted drawing locations instead of ground-truth locations | VersaGen · Sketchy | CLIP: 0.2524 -> 0.2542 | Helped |
| Add MCR (ablation) | ControlNet (SDXL) · COCO | ACC: 0.7075 -> 0.7284 | Helped |
| Remove MCR (ablation) | VersaGen · Text prompt and generated images | Prompt-image consistency | Hurt |
| Add MCR | VersaGen · COCO | FID: 0.3361 -> 0.3303 | Helped |
| Remove multi-object decoupling (ablation) | VersaGen · Multiple visual subjects | Generation quality | Hurt |
| Add multi-object decoupling (ablation) | VersaGen · Multiple visual subjects | Object overlap | Helped |
| Add adaptive control strength (ablation) | VersaGen · Low-quality user drawings | Generation quality | Helped |
| Add single visual subject control instead of text-only input (ablation) | VersaGen · COCO | CLIP: 0.2508 -> 0.2557 | Helped |
| Add multiple visual subjects instead of single-subject control (ablation) | VersaGen · COCO | CLIP: 0.2557 -> 0.2572 | Helped |
| Add scene background to multiple-subject control (ablation) | VersaGen · COCO | DINO: 0.7902 -> 0.8103 | Helped |
| Use VersaGen instead of Stable Diffusion | Text-to-image generation · COCO | CLIP: 0.2556 -> 0.2537 | Hurt |
Surprising findings
- Stable Diffusion achieves the highest CLIP score on COCO (0.2556), above VersaGen, which the authors attribute to multi-condition conflicts and edge-extraction limits.
- In the pilot study, participants spent more post-generation time with ControlNet than with plain Stable Diffusion, showing that good input conditions are hard to produce.
Limitations
- Training uses edge maps extracted from images as proxies for human free-hand sketches, which differ in quality.
- Localisation and conflict resolution depend on cross-attention maps, which may be inaccurate when prompts and drawings conflict.
- Inference adds test-time optimisation and multi-object decoupling hyperparameters (alpha, beta, lambda, a, b, c) that need tuning.
Experimental setup
Datasets: COCO Sketchy
Benchmarks: COCO Sketchy
Metrics: CLIP score FID DINO score ACC (Grounding DINO object accuracy) user study selection rate
Baselines: Stable Diffusion T2I-Adapter ControlNet UniControl GLIGEN InstanceDiffusion
Builds on: ControlNet Stable Diffusion Layout Guidance / cross-attention control
Closest papers by meaning
- MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis (AAAI 2025 · similarity 0.87)
- XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation (NeurIPS 2025 · similarity 0.87)
- Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation (CVPR 2025 · similarity 0.87)
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (ICLR 2024 · similarity 0.87)
- DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.