Research map / Image generation / Text-to-image diffusion
Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Suppresses negative-target content in text-to-image diffusion by SVD-based soft-weighted regularization of the text embedding matrix (including [EOT] embeddings) plus inference-time text embedding optimization with attention losses, requiring no fine-tuning.
Problem
Text-to-image diffusion models fail to suppress content explicitly requested to be omitted in the prompt (e.g., "a man without glasses" still produces glasses).
Approach
Analyzes the [EOT] padding embeddings of the CLIP text encoder as carrying redundant semantic information of the whole prompt, then applies soft-weighted shrinkage to singular values of a matrix formed from negative-target and [EOT] embeddings, followed by per-timestep optimization of the text embeddings with a positive-preservation loss and a negative-suppression loss.
Key results
- On random negative-target real-image editing, the method reaches DetScore 0.0384 and Clipscore 0.6857 versus 0.3381 and 0.7986 for the real image, the best among compared methods.
- On generated-image Car suppression, DetScore drops to 0.0419 vs 0.0935 for Concept-ablation (second best).
- On GQA-Inpaint, CLIP Acc is 92.8 vs 80.5 for Inst-Inpaint, though its FID (13.87) is worse than Inst-Inpaint (5.50).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add inference-time text embedding optimization with negative target suppression and positive target preservation losses to soft-weighted regularization (ablation) | Stable Diffusion v1.4 · Generated-image editing, random negative target | DetScore: 0.3668 -> 0.1321 | Helped |
| Add inference-time text embedding optimization with negative target suppression and positive target preservation losses to soft-weighted regularization (ablation) | Stable Diffusion v1.4 · Generated-image editing, random negative target | Clipscore: 0.7996 -> 0.6647 | Helped |
| Add inference-time text embedding optimization with negative target suppression and positive target preservation losses to soft-weighted regularization (ablation) | Stable Diffusion v1.4 · Generated-image editing, random negative target | IFID: 85.9 -> 176.4 | Helped |
| Add positive target prompt preservation loss (ablation) | SWR + negative target prompt suppression · Generated-image editing, random negative target | SSIM: 0.407 to 0.552 | Helped |
| Omit positive target prompt preservation loss (ablation) | SWR + negative target prompt suppression · Image editing | image structure preservation | Hurt |
| Use negative target suppression loss without positive target preservation loss (ablation) | SWR · Image editing | DetScore: 0.0692; IFID=242.3 | Mixed |
| Replace soft-weighted regularization with zeroing the top-K singular values (ablation) | Stable Diffusion · Negative target suppression | suppression and image quality | Mixed |
| Replace SVD-based extraction with a simple mean of the [EOT] embeddings (ablation) | Stable Diffusion · A man without glasses | DetScore: 0.1065 vs 0.6266 | Hurt |
| Add zeroing of the glasses token embedding to the mean [EOT] embedding (ablation) | Mean [EOT] embedding · A man without glasses | DetScore: 0.4892 | Hurt |
| Optimize value components directly instead of attention maps (ablation) | Inference-time text embedding optimization · Image editing | image quality | Hurt |
| Zero the target prompt attention map (ablation) | Stable Diffusion · Negative target suppression | target removal | Hurt |
| Increase the prompt length while zeroing both the target token and [EOT] embeddings (ablation) | Stable Diffusion · A man without glasses with added description words | DetScore: 0.084; 0.393; 0.455; 0.427 | Hurt |
Surprising findings
- Zeroing out the glasses token and all [EOT] embeddings removes glasses, yet zeroing only the [EOT] embeddings does not, showing [EOT] embeddings carry the negative-target information.
- Simple averaging of the [EOT] embeddings fails to extract the main semantic component (DetScore 0.6266).
Limitations
- Test-time optimization takes around half a minute, unsuitable for fast applications.
- Requires concise prompts; long descriptive prompts make suppression harder.
- Inst-Inpaint achieves better FID on GQA-Inpaint, and the method's FID is worse there (13.87).
Experimental setup
Datasets: COCO validation captions Unsplash images GQA-Inpaint ESD Van Gogh and Tyler Edlin prompts
Benchmarks: COCO captions generated-image suppression Unsplash and COCO real-image editing GQA-Inpaint
Metrics: CLIPScore IFID DetScore FID CLIP Accuracy
Baselines: Negative prompt P2P ESD Concept-ablation Forget-Me-Not Inst-Inpaint SEGA X-Decoder
Builds on: Prompt-to-Prompt Null-Text Inversion WNNM Negative prompt
Closest papers by meaning
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.91)
- Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models (NeurIPS 2025 · similarity 0.90)
- A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimization (NeurIPS 2024 · similarity 0.89)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.89)
- InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization (CVPR 2024 · similarity 0.89)
- Precise Parameter Localization for Textual Generation in Diffusion Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.