Research map / Image generation / Text-to-image diffusion
HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free text-guided image inpainting method (HD-Painter) that adds prompt-aware self-attention (PAIntA) and reweighted attention-score guidance (RASG) to Stable Inpainting, plus an inpainting-specific super-resolution step reaching 2K images.
Problem
Text-guided image inpainting models suffer from prompt neglect (background dominance and nearby object dominance), and existing methods struggle to scale to high resolution while staying prompt-faithful.
Approach
Replace self-attention with a prompt-aware variant that down-weights known-region tokens unrelated to the prompt (using cross-attention scores), and guide sampling with a cross-attention BCE objective via a reweighted gradient inserted into the general DDIM form so latents stay in-distribution; a known-region-blended super-resolution then upscales results to 2K.
Key results
- HD-Painter reaches CLIP score 26.25 and accuracy 67.59% on 10,000 MSCOCO samples, versus 25.62 and 59.02% for the DreamShaper Inpainting base.
- Aesthetic score 5.00 vs 4.96 for the base; PickScore favors HD-Painter over every baseline in at least 50% of comparisons (50.0% vs 51.98–58.69%).
- Ablation: PAIntA alone raises accuracy to 64.17% and RASG alone to 62.67%; combined gives 67.59%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RASG without PAIntA (ablation) | DreamShaper Inpainting · MSCOCO 2017 validation | CLIP score: 25.62 -> 25.82 | Helped |
| Add RASG without PAIntA (ablation) | DreamShaper Inpainting · MSCOCO 2017 validation | Accuracy: 59.02 % -> 62.67 % | Helped |
| Add RASG without PAIntA (ablation) | DreamShaper Inpainting · MSCOCO 2017 validation | Aesthetic score: 4.96 -> 4.97 | Helped |
| Add PAIntA without RASG (ablation) | DreamShaper Inpainting · MSCOCO 2017 validation | CLIP score: 25.62 -> 26.06 | Helped |
| Add PAIntA without RASG (ablation) | DreamShaper Inpainting · MSCOCO 2017 validation | Accuracy: 59.02 % -> 64.17 % | Helped |
| Add PAIntA without RASG (ablation) | DreamShaper Inpainting · MSCOCO 2017 validation | Aesthetic score: 4.96 -> 4.99 | Helped |
| Add RASG and PAIntA (ablation) | DreamShaper Inpainting · MSCOCO 2017 validation | CLIP score: 25.62 -> 26.25 | Helped |
| Add RASG and PAIntA (ablation) | DreamShaper Inpainting · MSCOCO 2017 validation | Accuracy: 59.02 % -> 67.59 % | Helped |
| Add RASG and PAIntA (ablation) | DreamShaper Inpainting · MSCOCO 2017 validation | Aesthetic score: 4.96 -> 5.00 | Helped |
Surprising findings
- Standard post-hoc guidance can push latents out of distribution, so the authors replace the stochastic DDIM noise term with a standardized gradient rather than adding it, which keeps the method hyperparameter-free.
Limitations
- Inherits quality limitations of the backbone model, such as extra limbs or illogical appearances.
- Relies on the backbone inpainting model's behavior; failure cases shown in appendix.
Experimental setup
Datasets: MSCOCO 2017 validation set Simulacra Aesthetic Captions (aesthetic predictor training, external)
Benchmarks: MSCOCO 2017 validation
Metrics: CLIP score generation class accuracy LAION aesthetic score PickScore
Baselines: GLIDE Blended Latent Diffusion (BLD) SDXL Inpainting Stable 2.0 Inpainting DreamShaper Inpainting DreamShaper-ControlNet Inpainting SmartBrush (reproduced)
Builds on: Stable Inpainting DDIM Attend-and-Excite (generative semantic nursing) Stable Super-Resolution Blended Latent Diffusion
Closest papers by meaning
- PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference (NeurIPS 2024 · similarity 0.89)
- Paint by Inpaint: Learning to Add Image Objects by Removing Them First (CVPR 2025 · similarity 0.89)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.88)
- One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting (NeurIPS 2025 · similarity 0.88)
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.88)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.