Research map / Image generation / Text-to-image diffusion

TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting

CVPR 2025 · Image generation › Text-to-image diffusion · approach: Compositional text-to-image guidance · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a BrushNet-style inpainting adapter directly on the few-step DMD2 text-to-image model using a 3-step adversarial scheme with a diffusion discriminator, yielding 4-step inpainting that beats multi-step BrushNet.

Problem

Diffusion-based image inpainting adapters require 20-50 sampling steps, which is costly; naively plugging adapters into few-step distilled models yields color shifts, oversaturation, and blur, and distilling a large BrushNet is memory- and compute-heavy.

Approach

Train an inpainting adapter directly on a few-step distilled model (DMD2) with a 3-step adversarial scheme: the slow SDXL-based adapter is trained with diffusion loss, the fast DMD2-based generator is trained with a diffusion-discriminator GAN loss plus background reconstruction loss, and the discriminator (with an assistant encoder/decoder and fake diffusion loss) is updated jointly.

Key results

• On DilationBench, TurboFill (4 steps) reaches mask-region Q-Align 4.5703 and CLIPIQA+ 0.7332, beating 50-step BrushNet* (4.4688, 0.7137).
• On HumanBench, TurboFill (4 steps) improves over 4-step BrushNet* on mask-region Q-Align (4.4727 vs 4.0508) and CLIPIQA+ (0.7257 vs 0.6327).
• Training cost is reported at 8 A100 GPUs for 50 hours, versus 8 V100 for 72 hours plus 64 A100 for 50 hours for the two-stage alternative, a reduction of over 10x in training time.

Measured effects reported in the paper

Surprising findings

• Directly inserting a multi-step BrushNet adapter into the 4-step DMD2 model degrades quality (oversaturation, lost detail) even though text alignment is preserved.
• 4-step TurboFill outperforms the 50-step BrushNet-Rand and BrushNet* on several image quality metrics.

Limitations

• PowerPaint V2 achieves the best scores on most metrics, exceeding TurboFill in the reported tables.
• Existing IQA metrics do not fully capture style mismatch between mask and background, so the paper relies on a user study in the supplementary material.

Experimental setup

Datasets: LocalCaptionData OpenImage V6 MSCOCO BrushBench

Benchmarks: DilationBench HumanBench

Metrics: Q-Align CLIPIQA+ TOPIQ CLIPIQA CLIP Sim

Baselines: BLD HD-Painter SDXL-Inpainting BrushNet-Rand BrushNet* PowerPaint V2

Builds on: BrushNet DMD2 ControlNet LCM SDXL

Closest papers by meaning

• MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting (CVPR 2025 · similarity 0.88)
• Paint by Inpaint: Learning to Add Image Objects by Removing Them First (CVPR 2025 · similarity 0.87)
• HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.87)
• PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference (NeurIPS 2024 · similarity 0.86)
• Reconstructing the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Inpainting Model (NeurIPS 2024 · similarity 0.86)
• Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.85)