Research map / Image generation / Text-to-image diffusion

Information Theoretic Text-to-Image Alignment

ICLR 2025 · Image generation › Text-to-image diffusion · approach: T2I evaluation and controllable adapters · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MI-TUNE, a self-supervised fine-tuning method that uses point-wise mutual information estimated from a diffusion model's own conditional/unconditional denoising difference to select synthetic prompt-image pairs and fine-tune DoRA adapters for better text-to-image alignment.

Problem

Text-to-image diffusion models often fail to follow prompts (attribute binding, spatial layout, complex prompts), and existing alignment methods need human annotation, auxiliary VQA/reward models, linguistic prompt analysis, or add inference overhead.

Approach

Estimate point-wise mutual information between prompt and image directly from the pre-trained diffusion denoiser (difference between conditional and unconditional noise predictions), use it to select top-k self-generated images per prompt as a fine-tuning set, and fine-tune lightweight DoRA adapters, requiring no auxiliary models or human labels.</1-2>

Key results

• MITUNE improves average BLIP alignment on T2I-CompBench from 45.81 (SD base) to 53.56 and average HPS from 26.17 to 27.21, outperforming all six alternative methods.
• In a user study, MITUNE was preferred in 48.35% average human selection vs 35.71% for SD base on T2I-CompBench.
• On SDXL, MITUNE raises average BLIP-based alignment gains of 5.5-14.6 points across categories over vanilla SDXL in several categories.

Measured effects reported in the paper

Surprising findings

• Fine-tuning on MI-selected images with few real images mixed in degrades BLIP alignment, although HPS sometimes improves.
• Selecting fine-tuning samples by HPS instead of MI degrades BLIP alignment performance.

Limitations

• Relies on pre-alignment of the base model; may strengthen but not induce alignment.
• Evaluation partly uses BLIP and HPS, which could bias selection if used for filtering; authors avoid BLIP in selection.
• User studies are small (42 to 102 surveys from few users).

Experimental setup

Datasets: T2I-CompBench DiffusionDB MS-COCO 2014

Benchmarks: T2I-CompBench

Metrics: BLIP score HPS human preference rate FID DINO CMMD UniDet

Baselines: Attend-and-Excite Structured Diffusion Guidance Syngen/SCG DPOK GORS HN SDXL

Builds on: Stable Diffusion DoRA T2I-CompBench DPOK GORS

Closest papers by meaning

• Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.92)
• Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.91)
• CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching (NeurIPS 2024 · similarity 0.90)
• Discriminative Probing and Tuning for Text-to-Image Generation (CVPR 2024 · similarity 0.90)
• Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
• Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences Through f-Divergence Minimization (AAAI 2025 · similarity 0.89)