Research map / Image generation / Text-to-image diffusion
MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTADiffusion, a BrushNet-style dual-branch inpainting diffusion model trained on a new 5M-image mask-text dataset (MTADataset) with an edge-prediction auxiliary loss and a VGG Gram-matrix style loss for object inpainting.
Problem
Text-guided object inpainting suffers from semantic misalignment between mask regions and prompts, structural distortion, and style inconsistency with the original image.
Approach
Annotate each object mask with detailed content and style descriptions via Grounded-SAM plus LLaVA (MTAPipeline) to build mask-text pairs, and combine joint edge prediction with a VGG Gram-matrix latent style loss on a BrushNet-like dual-branch architecture.
Key results
- On BrushBench, MTADiffusion reaches VQA score 68.97 vs. 68.22 for BrushNet and IR 12.69 vs. 12.52.
- On EditBench, CLIP similarity 29.12 vs. 28.87 for BrushNet, with IR 4.82 vs. 4.46.
- In a 30-participant user study, 66% of votes went to MTADiffusion for semantic alignment, 60% for structural stability, and 54% for style consistency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MTADiffusion instead of BrushNet | inpainting model · BrushBench | VQA Score ×102: 68.22 -> 68.97 | Helped |
| Use MTADiffusion instead of BrushNet | inpainting model · EditBench | CLIP Sim: 28.87 -> 29.12 | Helped |
| Use MTADiffusion instead of BrushNet | inpainting model · EditBench | PSNR: 33.66 -> 33.31 | Hurt |
| Use MTADiffusion instead of BrushNet | inpainting model · BrushBench | LPIPS ×103: 18.95 -> 18.94 | Helped |
| Train on MTADataset instead of BrushData (ablation) | BrushNet · BrushBench | IR ×10: 11.73 -> 12.08 | Helped |
| Train on MTADataset instead of BrushData (ablation) | BrushNet · BrushBench | CLIP Sim: 26.28 -> 26.41 | Helped |
| Use LLaVA detailed mask captions instead of original image captions (ablation) | BrushNet training · BrushBench | CLIP Sim: 26.31 -> 26.36 | Helped |
| Use LLaVA detailed mask captions instead of BLIP2 captions (ablation) | BrushNet training · BrushBench | IR ×10: 11.49 -> 11.76 | Helped |
| Use LLaVA detailed mask captions instead of Grounded-SAM labels (ablation) | BrushNet training · BrushBench | CLIP Sim: 26.23 -> 26.36 | Helped |
| Use LLaVA detailed mask captions instead of original image captions (ablation) | BrushNet training · BrushBench | IR ×10: 11.67 -> 11.76 | Helped |
| Use LLaVA detailed mask captions instead of BLIP2 captions (ablation) | BrushNet training · BrushBench | CLIP Sim: 26.28 -> 26.36 | Helped |
| Use LLaVA detailed mask captions instead of Grounded-SAM labels (ablation) | BrushNet training · BrushBench | IR ×10: 11.41 -> 11.76 | Helped |
Surprising findings
- Against BrushNet, the gains on masked-region reconstruction are negligible (PSNR 31.87 vs 31.82, LPIPS nearly identical) and on EditBench PSNR is slightly lower (33.31 vs 33.66).
Limitations
- Precision of text descriptions depends on the caption model; LLaVA sometimes gets object attributes such as color wrong.
- Ablations of the style and structure losses are deferred to the supplementary material and not in the main text.
- Gains over BrushNet are modest on several metrics, and the dataset is not released.
Experimental setup
Datasets: LAION MTADataset OpenImages v6 BrushData
Benchmarks: BrushBench EditBench
Metrics: ImageReward aesthetic score PSNR LPIPS MSE CLIP similarity VQA score user preference
Baselines: SDI CNI PowerPaint BrushNet BLD SmartBrush CAT-Diffusion
Builds on: BrushNet Stable Diffusion Grounded-SAM LLaVA ControlNet
Closest papers by meaning
- TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting (CVPR 2025 · similarity 0.88)
- Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.88)
- Paint by Inpaint: Learning to Add Image Objects by Removing Them First (CVPR 2025 · similarity 0.87)
- One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting (NeurIPS 2025 · similarity 0.87)
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.86)
- Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.