Research map / Image generation / Text-to-image diffusion
Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies token-level directions in tokenwise CLIP text embeddings, via contrastive prompts or diffusion-noise-based learning, to give continuous, subject-specific attribute control in T2I models without modifying the model.
Problem
Existing T2I control methods offer either subject-specific localization with coarse, discrete control or fine-grained continuous control applied globally to all subjects; no method offers both simultaneously.
Approach
Modulating the tokenwise CLIP embedding of a specific subject token along semantic directions (found by contrastive prompt differences or by backpropagating diffusion noise-prediction differences into the embedding space) yields continuous, subject-specific attribute control at zero inference cost.
Key results
- The learned directions achieve subject-specificity of 3.35 versus 4.14 for adding adjectives to the prompt and 3.93 for Prompt-to-Prompt, with LPIPS 0.10 per unit change, lower than Concept Sliders (0.20) and Prompt-to-Prompt (0.29).
- Combining the method with AdapEdit gives subject-specificity of 6.46 and LPIPS 0.05 (vs AdapEdit 6.92 and 0.10), with fully continuous control.
- The method adds zero inference cost (12.0s per image, same as the base model), while modulating after the first 20% of sampling steps improves disentanglement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a delay and apply modulation only after the first 20% of sampling steps (ablation) | Ours · Attribute control evaluation | LPIPS: 0.22 -> 0.10 | Helped |
| Add a delay and apply modulation only after the first 20% of sampling steps (ablation) | Ours · Attribute control evaluation | ∆Id: 0.50 -> 0.40 | Helped |
| Use learned directions instead of contrastive-prompt embedding differences (ablation) | Ours · Attribute control evaluation | LPIPS: 0.58 -> 0.10 | Helped |
| Use learned directions instead of contrastive-prompt embedding differences (ablation) | Ours · Attribute control evaluation | Subject-Specificity: 2.38 -> 3.35 | Helped |
| Distill the attribute direction into embedding space rather than directly modulating the noise-space direction with CFG (ablation) | Ours · Attribute control evaluation | LPIPS: 0.39 -> 0.10 | Helped |
| Distill the attribute direction into embedding space rather than directly modulating the noise-space direction with CFG (ablation) | Ours · Image generation | Time: 23.0s [2.17it/s] -> 12.0s [4.17it/s] | Helped |
| Combine attribute modulation with AdapEdit | Ours · Attribute control evaluation | LPIPS: 0.10 -> 0.05 | Helped |
| Combine attribute modulation with AdapEdit | Ours · Attribute control evaluation | ∆Id: 0.24 -> 0.19 | Helped |
| Combine attribute modulation with Prompt-to-Prompt | Ours · Attribute control evaluation | LPIPS: 0.29 -> 0.08 | Helped |
| Use continuous subject-specific attribute modulation rather than global Concept Sliders | Stable Diffusion XL · Multi-subject control | Subject-Specificity | Helped |
| Apply attribute modulation to subject tokens rather than changing text prompts | Stable Diffusion XL · Attribute expression control | attribute expression | Helped |
| Transfer learned modulation directions to models using the same text encoders without retraining | Stable Diffusion XL directions · Other image-generation models | attribute control | Helped |
Surprising findings
- The simple contrastive CLIP-difference direction causes unwanted side effects such as flipping a car's orientation, with much higher LPIPS (0.58) than the learned approach.
- Directions learned on SDXL transfer zero-shot to SDXL Turbo, SD 1.5, LDM3D, aMUSEd and PixArt-α without retraining.
Limitations
- Inherits the limitation that diffusion models sometimes mix up attributes between subjects.
- Linear modulations along CLIP token-embedding directions may entangle attributes; nonlinear modulations are not explored.
- Real image editing requires inversion, which depends on an off-the-shelf method and a caption.
Experimental setup
Metrics: subject-specificity ratio (delta CLIP) identity change (ReID cosine, Delta Id) LPIPS delta CLIP bidirectional generation time
Baselines: Concept Sliders Prompt-to-Prompt AdapEdit MasaCtrl ReNoise
Builds on: Concept Sliders Prompt-to-Prompt AdapEdit ReNoise Classifier-free guidance
Closest papers by meaning
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.90)
- Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function (NeurIPS 2024 · similarity 0.90)
- Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models (AAAI 2024 · similarity 0.89)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.88)
- Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.