Research map / Image generation / Text-to-image diffusion
Discriminative Probing and Tuning for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes Stable Diffusion's discriminative abilities (image-text matching and referring expression comprehension) with a lightweight Transformer adapter, then fine-tunes SD with LoRA on those objectives and adds discriminative-gradient self-correction at inference to improve text-image alignment.
Problem
Text-to-image models suffer from text-image misalignment (attribute binding, counting, relation confusion) due to weak compositional reasoning, and prior fixes rely on attention manipulation or LLM layout planning that depend on intermediate states.
Approach
Treat the discriminative ability of a T2I model as a proxy for its text-image alignment: attach a discriminative adapter (Transformer encoder-decoder with learnable queries) to frozen SD U-Net features to probe ITM and REC, then tune SD with LoRA on these discriminative objectives and use the adapter's gradients to steer denoising at inference (self-correction).
Key results
- On COCO-NSS1K (ID) with SD-v2.1, DPT+SC raises BLIP-M from 73.32 to 79.15 and CLIP from 34.96 to 35.75.
- On TIFA with SD-v2.1, DPT improves the score from 81.35 to 84.49 (DPT+SC 84.63).
- Discriminative tasks: DPT outperforms Diffusion Classifier and DiffusionITM by large margins on MSCOCO-HN and RefCOCO-family REC (e.g., DPT on RefCOCO val 52.73 vs DiffusionITM 28.88 zero-shot-generative setting).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| discriminatively tune with global matching (ablation) | Stable Diffusion-v2.1 · COCO-NSS1K (ID) | CLIP: 34.96 -> 35.14 | Helped |
| discriminatively tune with local grounding (ablation) | Stable Diffusion-v2.1 · COCO-NSS1K (ID) | CLIP: 34.96 -> 35.94 | Helped |
| combine global matching and local grounding objectives (ablation) | Stable Diffusion-v2.1 · COCO-NSS1K (ID) | CLIP: 34.96 -> 35.83 | Helped |
| combine global matching and local grounding objectives (ablation) | Stable Diffusion-v2.1 · CC-500 (OOD) | GLIP: 52.09 -> 53.29 | Mixed |
| add self-correction during inference (ablation) | DPT · CC-500 (OOD) | BLIP-M: 90.72 -> 91.33 | Helped |
| add self-correction during inference (ablation) | DPT · COCO-NSS1K (ID) | CLIP: 35.83 -> 35.75 | Hurt |
| add self-correction during inference (ablation) | DPT · TIFA | VQA accuracy: 84.49 -> 84.63 | Helped |
| add discriminative tuning | Stable Diffusion-v2.1 · COCO-NSS1K (ID) | CLIP: 34.96 -> 35.83 | Helped |
| add discriminative tuning | Stable Diffusion-v2.1 · CC-500 (OOD) | CLIP: 39.24 -> 40.23 | Helped |
Surprising findings
- Multi-task tuning (GM plus LG) slightly reduces some alignment metrics, suggesting contradictions between the two discriminative objectives.
- Probing deeper U-Net blocks improves generation but discriminative performance peaks at middle blocks and then deteriorates, a trade-off between generation and discrimination.
Limitations
- Gains on some metrics (e.g., GLIP, some CLIP scores on CC-500 for SD-v2.1) are marginal or negative.
- Multi-task discriminative tuning shows conflicts between matching and grounding objectives.
- Evaluated only on Stable Diffusion variants; generalization to other T2I models is not shown.
Experimental setup
Datasets: MSCOCO RefCOCO RefCOCO+ RefCOCOg
Benchmarks: COCO-NSS1K CC-500 ABC-6K TIFA T2I-CompBench MSCOCO-HN RefCOCO RefCOCO+ RefCOCOg
Metrics: CLIP score BLIP-ITM score BLIP-ITC score GLIP score IS FID VQA accuracy ITM accuracy REC accuracy
Baselines: Stable Diffusion-v1.4 Stable Diffusion-v2.1 LayoutLLM-T2I StructureDiffusion HN-DiffusionITM Attend-and-Excite VPGen LayoutGPT Diffusion Classifier CLIP
Builds on: Stable Diffusion DETR LoRA DiffusionITM Diffusion Classifier
Closest papers by meaning
- Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.91)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.91)
- Information Theoretic Text-to-Image Alignment (ICLR 2025 · similarity 0.90)
- Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.90)
- Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences Through f-Divergence Minimization (AAAI 2025 · similarity 0.89)
- CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.