AI Paper Atlas

Research map / Image generation / Text-to-image diffusion

Ablating Concepts in Text-to-Image Diffusion Models

Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, Jun-Yan Zhu

ICCV 2023 · Image generation › Text-to-image diffusion · approach: Text-driven synthesis and semantic guidance · 211 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune the full U-Net weights with the model-based ablation objective on memorized promptsStable Diffusion · Memorized images (8 prompts)% samples with SSCD similarity >= 0.5: 60.1 -> 0.3Improved
ablate concepts with our method versus MSCOCO FID of the pretrained modelStable Diffusion · MSCOCO FIDFID: 16.35 -> 16.99 ± 0.2No clear effect
use model-based objective (match model predictions) instead of loss maximization baseline (ablation)Concept ablation, cross-attention fine-tuning · Instances (Grumpy Cat, Snoopy, Nemo, R2D2)CLIP Score / accuracy on surrounding conceptsImproved
model-based variant instead of noise-based variant (ablation)Concept ablation · Instancesconvergence speed / CLIP metricsImproved
fine-tune text embedding only versus cross-attention (ablation)Concept ablation · Instances (spelling-mistake prompts)robustness to spelling mistakesHurt
fine-tune full weights instead of other parameter subsets for memorized images (ablation)Concept ablation · Memorized imagesmemorization removalImproved
use reverse KL approximation instead of forward KL (ablation)Concept ablation · Styles vs instancesCLIP metricsMixed
choose broader anchor category (British Shorthair Cat or Felidae) instead of Cat (ablation)Grumpy Cat ablation · Grumpy Catqualitative samplesNo clear effect
train on 1000 images instead of 200 (ablation)Concept ablation · Surrounding conceptsCLIP metricsNo clear effect
increase training images from 1000 to 10k (ablation)Concept ablation · Grumpy Catablation performanceNo clear effect
increase number of unique training prompts (10, 50 vs 200) (ablation)Concept ablation · Grumpy Catconvergence speedImproved
replace ChatGPT with QLoRA for generating training prompts (ablation)Grumpy Cat ablation · Grumpy Cat target and surroundingCLIP Score: 0.639 -> 0.651 (target); 0.796 -> 0.801 (surrounding)No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy