Research map / Image generation / Text-to-image diffusion
Precise Parameter Localization for Textual Generation in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Localizes the few cross/joint attention layers (<1% of parameters) that control visual text generation in diffusion models via activation patching, then uses them for LoRA fine-tuning, text editing, and cost-free toxic text prevention.
Problem
Diffusion models render high-quality text in images, but it is unclear which internal components control textual content; this matters for improving text generation, editing text in images, and preventing toxic text generation.
Approach
Activation patching of text keys and values from a target prompt into individual cross/joint attention layers of a source-prompt generation identifies a tiny set of layers (1-3 layers) that determine the rendered text; these layers are then used for targeted LoRA fine-tuning, image text editing, and substituting toxic words on the fly.
Key results
- Only 3 of 70 cross-attention layers (0.61% of SDXL parameters), 1 of 22 layers in DeepFloyd IF (0.21%) and 1 of 24 in SD3 (0.23%) govern text content.
- LoRA on the three localized SDXL layers improves OCR F1 and keeps recall/precision, whereas LoRA on all cross-attention layers collapses diversity after about 20 epochs.
- Localized patching edits text in images with higher SSIM/PSNR and lower MSE than P2P in most settings, and is much faster (about 10 s vs 31 s for SDXL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LoRA on localized attention layers rather than all cross-attention layers | SDXL · Text generation fine-tuning | OCR F1 | Helped |
| Patch the target prompt into localized layers instead of using P2P | SDXL · SimpleBench and CreativeBench | OCR F1: 0.29 -> 0.43; 0.26 -> 0.33 | Helped |
| Patch localized layers instead of P2P* | SD3 · SimpleBench | OCR F1: 0.90 -> 0.68 | Hurt |
| Patch localized layers instead of P2P* | SD3 · CreativeBench | OCR F1: 0.66 -> 0.39 | Hurt |
| Patch from timestep t_s=48 instead of t_s=50 (ablation) | DeepFloyd IF · SimpleBench | OCR F1: 0.70 -> 0.70 | No clear effect |
| Increase injected layers from 1 to 2 (ablation) | SDXL text editing · SDXL text editing | OCR F1 (Text_T): 0.20 -> 0.30 | Helped |
| Increase injected layers from 3 to 70 (ablation) | SDXL text editing · SDXL text editing | SSIM: 0.90 -> 0.72 | Hurt |
| Increase injected layers from 3 to 70 (ablation) | SDXL text editing · SDXL text editing | OCR F1 (Text_T): 0.32 -> 0.34 | Helped |
| Fine-tune LoRA on the localized three layers rather than other sets of three layers (ablation) | SDXL · Text generation fine-tuning | OCR F1 | Helped |
| Replace toxic text only in localized layers rather than swapping the prompt for the whole model | SDXL · Toxic text prevention | MSE: 79.41 -> 48.20 | Helped |
| Replace toxic text only in localized layers rather than swapping the prompt for the whole model | SDXL · Toxic text prevention | Toxicity score: 0.000 -> 0.003 | Hurt |
| Patch the target prompt from a later diffusion timestep (ablation) | SDXL, DeepFloyd IF, and SD3 · Text editing | Image and text alignment | Helped |
Surprising findings
- Patching only one or three layers controls the rendered text while the other visual content of the image is preserved, and the layers ignore template changes.
- The text style (e.g., handwritten, neon) is not controlled by the same layer as the text content; at least 7 SD3 layers are needed to change style.
Limitations
- Toxic-text prevention relies on an LLM to rephrase the toxic word, and prompt swap slightly outperforms it in toxicity removal.
- Editing quality depends on the base model's text generation quality; longer texts produce errors such as omitted letters.
- Style control is distributed over multiple layers and cannot be isolated to one layer.
Experimental setup
Datasets: MARIO-LAION 10M SimpleBench CreativeBench GlyphDraw prompts Wikipedia keyword pool toxic word list
Benchmarks: SimpleBench CreativeBench
Metrics: OCR F1 CLIP-T Levenshtein distance MSE SSIM PSNR precision recall toxicity score execution time
Baselines: Prompt-to-Prompt (P2P) P2P* LoRA on all cross-attention layers Negative prompt Safe Diffusion Prompt Swap
Builds on: activation patching prompt-to-prompt LoRA Safe Diffusion
Closest papers by meaning
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.90)
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.90)
- InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization (CVPR 2024 · similarity 0.89)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.89)
- On Mechanistic Knowledge Localization in Text-to-Image Generative Models (ICML 2024 · similarity 0.89)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.