Research map / Image generation / Text-to-image diffusion
Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Diptych Prompting, a training-free zero-shot subject-driven generator that reframes subject-driven T2I as diptych inpainting with FLUX.1-dev plus a ControlNet-Inpainting module, using background removal and reference attention enhancement.
Problem
Subject-driven text-to-image generation: producing images of a new subject in a text-specified context without per-subject fine-tuning, while keeping fine-grained subject alignment that encoder-based image prompting methods often lack.
Approach
Arrange an incomplete diptych with the segmented reference subject in the left panel and inpaint the right panel under a diptych text prompt using FLUX with ControlNet inpainting, and boost attention from right-panel queries to left-panel keys (reference attention enhancement) to preserve details.
Key results
- On DreamBench, Diptych Prompting (FLUX) reaches DINO 0.689, CLIP-I 0.758, CLIP-T 0.344, versus IP-Adapter (FLUX) at 0.561/0.725/0.351 and MS-Diffusion at 0.671/0.792/0.321.
- In a human study with 150 participants, Diptych Prompting wins subject alignment against ELITE (77.9%), BLIP-Diffusion (73.8%), and λ-Eclipse (80.4%) and is significantly preferred (p<0.01).
- Removing background segmentation raises DINO to 0.759 but lowers CLIP-T to 0.333, indicating copy-and-paste leakage.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Diptych Prompting instead of IP-Adapter | DreamBench subject-driven generation · DreamBench | DINO: 0.561 -> 0.689 | Helped |
| use Diptych Prompting instead of IP-Adapter | DreamBench subject-driven generation · DreamBench | CLIP-T: 0.351 -> 0.344 | Hurt |
| add background removal (ablation) | Diptych Prompting · DreamBench | CLIP-T: 0.333 -> 0.344 | Helped |
| add background removal (ablation) | Diptych Prompting · DreamBench | DINO: 0.759 -> 0.689 | Hurt |
| increase reference attention rescaling from λ=1.0 to λ=1.3 (ablation) | Diptych Prompting · DreamBench | DINO: 0.647 -> 0.689 | Helped |
| increase reference attention rescaling from λ=1.0 to λ=1.3 (ablation) | Diptych Prompting · DreamBench | CLIP-T: 0.343 -> 0.344 | No clear effect |
| increase reference attention rescaling from λ=1.3 to λ=1.5 (ablation) | Diptych Prompting · DreamBench | DINO: 0.689 -> 0.670 | Hurt |
| replace zero-shot inpainting with ControlNet inpainting (ablation) | FLUX Diptych Prompting · DreamBench | DINO: 0.555 -> 0.689 | Helped |
| replace SD-3 with FLUX as the base model (ablation) | Diptych Prompting with ControlNet scale 0.95 · DreamBench | DINO: 0.576 -> 0.689 | Helped |
| increase ControlNet conditioning scale from 0.5 to 0.8 (ablation) | FLUX Diptych Prompting · DreamBench | DINO: 0.628 -> 0.670 | Helped |
| increase ControlNet conditioning scale from 0.8 to 0.95 (ablation) | FLUX Diptych Prompting · DreamBench | DINO: 0.670 -> 0.689 | Helped |
Surprising findings
- Removing background from the reference raises subject metrics but causes copy-paste-like outputs that hurt text alignment.
- Zero-shot diptych inpainting with SD-3 performs poorly, suggesting only FLUX produces accurate diptychs among tested models.
Limitations
- Slight disadvantage in CLIP-I compared with encoder-based methods that use CLIP image encoders.
- Relies on the capability of a large-scale TTI model (FLUX) and its inpainting ControlNet; smaller models fail to produce accurate diptychs.
Experimental setup
Datasets: DreamBench
Benchmarks: DreamBench
Metrics: DINO CLIP-I CLIP-T human preference win rate
Baselines: ELITE BLIP-Diffusion Kosmos-G Subject-Diffusion IP-Adapter MS-Diffusion λ-Eclipse
Builds on: FLUX ControlNet Grounding DINO Segment Anything Model HQ-Edit
Closest papers by meaning
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.88)
- MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance (ICLR 2025 · similarity 0.88)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.88)
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.88)
- Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis (CVPR 2024 · similarity 0.87)
- PreciseCam: Precise Camera Control for Text-to-Image Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.