Research map / Image generation / Text-to-image diffusion
Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DATE, a training-free method that updates text embeddings at each diffusion sampling step via a normalized gradient of a text-image evaluation function (CLIP score or ImageReward) on the predicted clean image, improving text-image alignment for Stable Diffusion and other T2I models.
Problem
Text-to-image diffusion models use fixed text embeddings across all sampling timesteps, which limits adaptation to the evolving generation process and can yield weaker text-image alignment.
Approach
At each sampling step, a single normalized-gradient step (sharpness-aware-style, first-order Taylor with L2 ball radius rho) adjusts the text embedding to maximize an alignment score evaluated on the Tweedie mean predicted image, so the conditioning adapts per timestep and per instance without retraining.
Key results
- On SD v1.5 with DDIM (50 steps), DATE with all updates using ImageReward raises ImageReward from 0.2132 (fixed) to 1.2972, while CLIP score rises from 0.3204 to 0.3292 with CLIP-score updates and FID drops from 18.66 to 17.22.
- On PixArt-alpha with DPM-Solver (20 steps), DATE with 50% updates using ImageReward improves ImageReward from 0.8140 to 0.9514 at comparable sampling time.
- The paper reports DATE outperforms fixed embeddings across SD3, FLUX, and SDXL backbones and improves multi-concept generation (AnE) and text-guided editing (DDPM inversion) on LPIPS-CLIP trade-off.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt text embeddings at sampling steps using the CLIP score objective | SD v1.5 with DDIM · COCO validation | CLIP score: 0.3204 -> 0.3292 | Helped |
| Adapt text embeddings at sampling steps using the ImageReward objective | SD v1.5 with DDIM · COCO validation | ImageReward: 0.2132 -> 1.2972 | Helped |
| Replace the embedding gradient with a random Gaussian vector (ablation) | DATE · Text-to-image generation | text-image alignment | No clear effect |
| Align the prompt with perturbed data using h(x_t;y) instead of the mean predicted image (ablation) | DATE · Text-to-image generation | generation performance | Hurt |
| Use an unnormalized embedding gradient (ablation) | DATE · Text-to-image generation | text-image alignment | Hurt |
| Increase the update scale rho (ablation) | DATE · Text-to-image generation | generation performance | Mixed |
| Initialize each update from the previously updated embedding instead of the pretrained encoder output (ablation) | DATE · COCO validation | CLIP score | Helped |
| Increase the proportion of sampling steps with embedding updates (ablation) | DATE · Text-to-image generation | generation performance | Helped |
| Schedule updates at middle-to-late rather than early sampling steps (ablation) | DATE · Text-to-image generation | text-image alignment | Helped |
Surprising findings
- Using ImageReward as the update objective on SD v1.5 with all updates lifts ImageReward far above baseline (1.30 vs 0.21) while CLIP score changes only slightly, indicating the objective strongly shifts the targeted metric.
- Random Gaussian updates perform similarly to the fixed embedding, while aligning the prompt with the perturbed noisy image degrades performance.
Limitations
- Gradient computation through diffusion and evaluation networks substantially increases sampling time (about 4x for all-step updates)
- Taylor approximation error grows with the radius rho; higher rho degrades performance in some regions
- Convexity assumption for the Jensen bound may not hold since evaluation functions are neural networks
Experimental setup
Datasets: COCO validation set AnE dataset ImageNet-R-TI2I
Benchmarks: COCO validation (5,000 images) AnE multi-concept prompts ImageNet-R-TI2I image editing
Metrics: zero-shot FID CLIP score ImageReward LPIPS PickScore Aesthetic Score full prompt similarity minimum object similarity
Baselines: Fixed text embedding EBCA Universal Guidance CONFORM DDPM inversion
Builds on: Universal Guidance Sharpness-aware minimization DDIM Classifier guidance DDPM inversion
Closest papers by meaning
- Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.90)
- A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimization (NeurIPS 2024 · similarity 0.90)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.89)
- Text Diffusion with Reinforced Conditioning (AAAI 2024 · similarity 0.89)
- Precise Parameter Localization for Textual Generation in Diffusion Models (ICLR 2025 · similarity 0.89)
- LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.