Research map / Image generation / Text-to-image diffusion
TextCraftor: Your Text Encoder Can be Image Quality Controller
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes the CLIP text encoder of Stable Diffusion v1.5 end-to-end through differentiable reward models (aesthetics, PickScore, HPSv2) on prompts only, with a CLIP-similarity constraint, improving image quality and text-image alignment without extra inference cost.
Problem
Stable Diffusion often produces images that do not align well with prompts and needs multiple runs or prompt engineering; prior fixes fine-tune the UNet on paired data or swap in larger text encoders, which add cost.
Approach
Fine-tune only the pretrained CLIP text encoder by backpropagating differentiable reward scores (computed on images from a fixed-step DDIM rollout on prompts alone), and keep the CLIP image-text cosine similarity as an always-on constraint to prevent mode collapse and preserve generality.
Key results
- On Parti-Prompts, TextCraftor (Text+UNet) with all rewards reaches Aesthetics 6.4166, PickScore 19.479, HPSv2 0.2900, versus SDv1.5 at 5.2634, 18.834, 0.2703.
- On the HPSv2 benchmark, TextCraftor (HPSv2) averages 0.2910 versus 0.2688 for SDv1.5 and 0.2773 for SDXL Base 0.9.
- In human evaluation on Parti-Prompts, TextCraftor is preferred over SDv1.5 at a 71.7% win rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune the text encoder with combined rewards | SDv1.5 · Parti-Prompts (1632) | PickScore: 18.834 -> 19.157 | Helped |
| fine-tune the text encoder with combined rewards | SDv1.5 · HPSv2 benchmark | Average HPSv2: 0.2688 -> 0.2807 | Helped |
| add UNet reward fine-tuning after text-encoder fine-tuning (ablation) | TextCraftor · Parti-Prompts (1632) | PickScore: 19.157 -> 19.479 | Helped |
| add UNet reward fine-tuning after text-encoder fine-tuning (ablation) | TextCraftor (HPSv2) · HPSv2 benchmark | Average HPSv2: 0.2910 -> 0.2983 | Helped |
| use Aesthetics reward for text-encoder fine-tuning (ablation) | SDv1.5 · Parti-Prompts (1632) | Aesthetics: 5.2634 -> 5.5212 | Helped |
| use Aesthetics reward for text-encoder fine-tuning (ablation) | SDv1.5 · Parti-Prompts (1632) | HPSv2: 0.2703 -> 0.2670 | Hurt |
| use PickScore reward for text-encoder fine-tuning (ablation) | SDv1.5 · Parti-Prompts (1632) | PickScore: 18.834 -> 19.023 | Helped |
| use PickScore reward for text-encoder fine-tuning (ablation) | SDv1.5 · Parti-Prompts (1632) | HPSv2: 0.2703 -> 0.2641 | Hurt |
| use HPSv2 reward for text-encoder fine-tuning (ablation) | SDv1.5 · Parti-Prompts (1632) | HPSv2: 0.2703 -> 0.2800 | Helped |
| use direct fine-tuning rather than prompt-based reward fine-tuning (ablation) | SDv1.5 · Parti-Prompts (1632) | PickScore: 18.834 -> 18.750 | No clear effect |
| add the CLIP similarity constraint during reward fine-tuning (ablation) | TextCraftor · Parti-Prompts | generality | Helped |
| use prompt engineering instead of TextCraftor | SDv1.5 · Parti-Prompts (1632) | PickScore: 18.834 -> 17.311 | Hurt |
Surprising findings
- Direct fine-tuning of the text encoder with reward on predicted x0 did not give reliable improvements, motivating the prompt-based rollout approach.
- Prompt engineering lowered text-image alignment scores (HPSv2 0.2599 vs 0.2703 for SDv1.5) despite higher aesthetics.
Limitations
- Relies on public reward models whose biases (e.g., toward artistic, saturated styles) are transferred to generations.
- Direct fine-tuning on predicted x0 is unreliable at noisy timesteps, so the method depends on a fixed 25-step DDIM rollout with gradient checkpointing.
Experimental setup
Datasets: OpenPrompt LAION-2B Parti-Prompts HPSv2 benchmark
Benchmarks: Parti-Prompts HPSv2 benchmark
Metrics: Aesthetics score PickScore HPSv2 score human win rate
Baselines: SDv1.5 SDv2.0 SDXL Base 0.9 DeepFloyd-XL DDPO Prompt Engineering Direct Fine-tune
Builds on: Stable Diffusion DDPO CLIP DDIM
Closest papers by meaning
- Scaling Down Text Encoders of Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.89)
- One-Way Ticket: Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.88)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.88)
- Precise Parameter Localization for Textual Generation in Diffusion Models (ICLR 2025 · similarity 0.88)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.87)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.