Research map / Image generation / Text-to-image diffusion
Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the TIA2 benchmark for evaluating text-image alignment reward models and TextNorm, a softmax-based confidence calibration of rewards over semantically contrastive prompts, showing it reduces reward overoptimization in SD v2.1 fine-tuning.
Problem
Reward models trained on human feedback are imperfect proxies for text-image alignment, and excessive optimization against them (reward overoptimization) degrades fine-tuned text-to-image models, especially when the reward is poorly aligned with humans.
Approach
Calibrate a reward by softmax-normalizing it against rewards for syntactically similar but semantically contrastive prompts (generated by rules or LLMs), using relative reward comparisons as a confidence measure; optionally ensemble normalized rewards from multiple reward models with a variance penalty.
Key results
- On the TIA2 comprehensive set, TextNorm achieves AUROC 0.788 and AUPRC 0.727 versus 0.747 / 0.675 for PickScore.
- On the counting set, TextNorm reaches AUROC 0.807 vs 0.731 for PickScore.
- Human evaluation reports roughly twice as many wins for text-image alignment with TextNorm-based SFT compared to baseline reward models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TextNorm using contrastive prompts and the reward-model ensemble | PickScore and ImageReward · TIA2 C100 | AUROC: 0.761 -> 0.788 | Helped |
| Apply TextNorm using contrastive prompts and the reward-model ensemble | PickScore and ImageReward · TIA2 C150 | AUROC: 0.731 -> 0.807 | Helped |
| Apply TextNorm using contrastive prompts and the reward-model ensemble | PickScore and ImageReward · TIA2 C300 | AUROC: 0.785 -> 0.844 | Helped |
| Use random rather than suitable contrastive prompts (ablation) | PickScore with TextNorm · TIA2 benchmark average | AUROC: 0.764 -> 0.641 | Hurt |
| Use LLM-generated contrastive prompts (ablation) | PickScore with TextNorm · TIA2 benchmark average | AUROC: 0.764 -> 0.780 | Helped |
| Add an ensemble to LLM-prompt TextNorm (ablation) | PickScore with TextNorm · TIA2 benchmark average | AUROC: 0.780 -> 0.822 | Helped |
| Fine-tune with TextNorm rewards instead of baseline reward models | Stable Diffusion v2.1 using SFT · Human evaluation, text-image alignment | win rate: twice as many wins | Helped |
| Fine-tune with TextNorm rewards instead of baseline reward models | Stable Diffusion v2.1 using RL · Human evaluation, text-image alignment and image quality | human preference | Mixed |
Surprising findings
- BLIP-2 quality for best-of-n selection degrades as n increases, showing overoptimization with a poorly aligned reward.
- TextNorm RL fine-tuning improves text-image alignment at the expense of image quality.
Limitations
- Relies on external LLM (ChatGPT) and VLM components whose quality strongly affects results.
- Ensemble weighting and temperature were only lightly tuned on the benchmark's AP metrics.
- RL fine-tuning with TextNorm reduced image quality while improving alignment.
Experimental setup
Datasets: DrawBench PartiPrompt ImageRewardDB Pick-a-Pic CIFAR-10 MS-COCO
Benchmarks: TIA2
Metrics: AUROC AUPRC AP@5 AP@10 AP@25 Spearman Kendall human win/tie/loss rate
Baselines: CLIP BLIP-2 ImageReward PickScore
Builds on: DDPO ImageReward PickScore LoRA
Closest papers by meaning
- Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation (CVPR 2025 · similarity 0.90)
- Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.88)
- Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences Through f-Divergence Minimization (AAAI 2025 · similarity 0.88)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.88)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.88)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.