Research map / Image generation / Text-to-image diffusion
One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NTN-Diff, a training-free null-text-null frequency-aware diffusion scheme that swaps DCT low/mid frequency bands across null-text and text-guided denoising branches to achieve text-guided inpainting with both unmasked preservation and masked/unmasked semantic consistency.
Problem
Text-guided image inpainting struggles to jointly preserve unmasked regions and keep semantics consistent between unmasked and inpainted masked regions; prior diffusion methods fix one at the expense of the other.
Approach
Decompose semantic consistency into per-frequency-band consistency: run null-text and text-guided denoising branches in the early stage, substituting DCT low-frequency and mid-frequency bands across branches (mid-frequency is stable under text prompts), then a final text-guided late stage with unmasked-region substitution.
Key results
- On BrushBench inside inpainting, NTN-Diff* reaches PSNR 40.70, MSE 0.11e-3, LPIPS 0.88e-3 versus BrushNet* at 31.94, 0.80e-3, 18.67e-3.
- On EditBench, NTN-Diff* achieves PSNR 36.83 vs 33.66 for BrushNet* and CLIP Score 29.72 vs 28.87 (BrushEdit 29.62).
- Ablating the early-stage null-text/text-guided branches (Case C) reduces performance, with NTN-Diff gaining up to 44.8% over Case C.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NTN-Diff instead of the strongest reported baseline for each BrushBench inside-inpainting metric | Text-guided inpainting with Stable Diffusion v1.5 · BrushBench inside inpainting | IR, HPS v2, PSNR, MSE, LPIPS, and CLIP Score: 12.36 -> 12.45 IR; 27.40 -> 27.57 HPS v2; 22.61 -> 23.51 PSNR; 9.31 -> 6.50 MSE; 43.50 -> 40.79 LPIPS; 26.48 -> 26.54 CL | Helped |
| Use NTN-Diff instead of BrushNet | Text-guided inpainting with Stable Diffusion v1.5 · BrushBench outside inpainting | IR, HPS v2, PSNR, MSE, LPIPS, and CLIP Score: 10.82 -> 11.54 IR; 28.02 -> 28.22 HPS v2; 18.06 -> 18.47 PSNR; 22.86 -> 20.44 MSE; 15.08 -> 14.46 LPIPS; 27.33 -> 27.54 | Helped |
| Add BrushNet blending operation to NTN-Diff | NTN-Diff · EditBench inpainting | PSNR, MSE, LPIPS, SSIM, and CLIP Score: 33.66 -> 36.83 PSNR; 0.63 -> 0.28 MSE; 10.12 -> 5.39 LPIPS; 98.13 -> 99.46 SSIM; 28.87 -> 29.72 CLIP Score | Helped |
| Remove masked self-attention from the first null-text process (ablation) | NTN-Diff · EditBench and BrushBench | IR, PSNR, LPIPS, and CLIP Score: EditBench IR 3.10 -> 2.41; PSNR 22.65 -> 22.36; LPIPS 24.21 -> 26.60; CLIP Score 28.95 -> 28.62; BrushBench IR 11.12 -> | Hurt |
| Remove both early null-text and text-guided denoising processes (ablation) | NTN-Diff · EditBench and BrushBench | IR, PSNR, LPIPS, and CLIP Score: EditBench IR 3.10 -> 1.24; PSNR 22.65 -> 22.39; LPIPS 24.21 -> 24.82; CLIP Score 28.95 -> 28.53; BrushBench IR 11.12 -> | Hurt |
| Remove the early null-text denoising process (ablation) | NTN-Diff · EditBench and BrushBench | IR, PSNR, LPIPS, and CLIP Score: EditBench IR 3.10 -> 2.14; PSNR 22.65 -> 22.53; LPIPS 24.21 -> 25.24; CLIP Score 28.95 -> 28.60; BrushBench IR 11.12 -> | Hurt |
| Set λ to 0.9 instead of 0.6 (ablation) | NTN-Diff · EditBench and BrushBench | IR, PSNR, LPIPS, and CLIP Score: EditBench IR 3.10 -> 2.41; PSNR 22.65 -> 22.03; LPIPS 24.21 -> 30.10; CLIP Score 28.95 -> 28.89; BrushBench IR 11.12 -> | Mixed |
| Set λ to 0.8 instead of 0.6 (ablation) | NTN-Diff · EditBench and BrushBench | IR, PSNR, LPIPS, and CLIP Score: EditBench IR 3.10 -> 2.67; PSNR 22.65 -> 22.19; LPIPS 24.21 -> 29.28; CLIP Score 28.95 -> 28.76; BrushBench IR 11.12 -> | Mixed |
| Set λ to 0.7 instead of 0.6 (ablation) | NTN-Diff · EditBench and BrushBench | IR, PSNR, LPIPS, and CLIP Score: EditBench IR 3.10 -> 2.78; PSNR 22.65 -> 22.44; LPIPS 24.21 -> 26.95; CLIP Score 28.95 -> 29.20; BrushBench IR 11.12 -> | Mixed |
| Set λ to 0.5 instead of 0.6 (ablation) | NTN-Diff · EditBench and BrushBench | IR, PSNR, LPIPS, and CLIP Score: EditBench IR 3.10 -> 1.10; PSNR 22.65 -> 22.63; LPIPS 24.21 -> 23.16; CLIP Score 28.95 -> 28.81; BrushBench IR 11.12 -> | Mixed |
| Replace the first null-text process with text-guided denoising (TTN-Diff) (ablation) | NTN-Diff · EditBench and BrushBench | IR, PSNR, LPIPS, and CLIP Score: EditBench IR 3.10 -> 1.57; PSNR 22.65 -> 22.57; LPIPS 24.21 -> 25.12; CLIP Score 28.95 -> 28.87; BrushBench IR 11.12 -> | Hurt |
| Replace the last null-text process with text-guided denoising (NTT-Diff) (ablation) | NTN-Diff · EditBench and BrushBench | IR, PSNR, LPIPS, and CLIP Score: EditBench IR 3.10 -> 2.12; PSNR 22.65 -> 22.58; LPIPS 24.21 -> 24.99; CLIP Score 28.95 -> 28.89; BrushBench IR 11.12 -> | Hurt |
Surprising findings
- Mid-frequency DCT band is reported to stabilize quickly under text prompts and is less disturbed than low-frequency band in the early denoising stage.
- The method reports lowest VRAM (2.11 GB) among compared methods despite three parallel denoising branches in the early stage.
Limitations
- Three parallel null-text-null denoising processes in the early stage incur larger computational cost than a single process.
- Hyperparameters (lambda, frequency thresholds) must be tuned; performance is sensitive to lambda (e.g., 0.9 worst, 0.6 best).
Experimental setup
Datasets: BrushBench EditBench
Benchmarks: BrushBench EditBench
Metrics: Image Reward HPS v2 PSNR MSE LPIPS CLIP Score SSIM
Baselines: BLD CNI PP BrushNet HDP SDI BrushEdit
Builds on: Stable Diffusion BLD BrushNet DDPM
Closest papers by meaning
- Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation (NeurIPS 2026 · similarity 0.90)
- InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization (CVPR 2024 · similarity 0.89)
- Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.89)
- Text Image Inpainting via Global Structure-Guided Diffusion Models (AAAI 2024 · similarity 0.89)
- Disentangling Dual Image References in Frequency Aware Diffusion Models for Personalized Generation (NeurIPS 2026 · similarity 0.88)
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.