Research map / Image generation / Text-to-image diffusion
AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AGFSync, a fully AI-driven pipeline where LLM-generated prompts, multi-aspect VLM/CLIP/aesthetic scores build preference pairs that are used to align SD v1.4, SD v1.5 and SDXL-base via Diffusion-DPO.
Problem
Text-to-image diffusion models struggle with prompt following and image quality, and preference-alignment methods depend on costly human-labeled data and single-aspect quality evaluation.
Approach
Build a human-free preference dataset: LLM generates diverse captions and QA pairs, diffusion models generate 8 candidates per prompt, and a weighted sum of VQA, CLIP and aesthetic scores picks best and worst images as winner/loser for Diffusion-DPO, without thresholds.
Key results
- On TIFA, AGFSync raises SD v1.4 VQA score from 76.6 to 77.9 and aesthetic score from 44.6 to 47.9, versus DreamSync's +1.3 sum.
- On TIFA, AGFSync improves SDXL-base VQA score 82.0 to 83.3 and aesthetic 60.9 to 65.2 (sum +5.5).
- GPT-4V win rates vs original models average 62%, 67%, and 69% for SD v1.4, SD v1.5, and SDXL-base respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply AGFSync DPO alignment | SD v1.4 · TIFA | VQA score: 76.6 -> 77.9 (+1.3) | Helped |
| Apply AGFSync DPO alignment | SD v1.4 · TIFA | aesthetic score: 44.6 -> 47.9 (+3.3) | Helped |
| Apply AGFSync DPO alignment | SD v1.5 · TIFA | VQA score: 77.1 -> 78.7 (+1.6) | Helped |
| Apply AGFSync DPO alignment | SD v1.5 · TIFA | aesthetic score: 48.0 -> 49.1 (+1.1) | Helped |
| Apply AGFSync DPO alignment | SDXL-base · TIFA | VQA score: 82.0 -> 83.3 (+1.3) | Helped |
| Apply AGFSync DPO alignment | SDXL-base · TIFA | aesthetic score: 60.9 -> 65.2 (+4.3) | Helped |
| Apply AGFSync instead of MJHQ-30K+SFT | SDXL-base · TIFA | aesthetic score: 61.1 -> 65.2 | Helped |
| Use all three scoring dimensions: CLIP, VQA, and aesthetic (ablation) | SD v1.5 preference construction · TIFA | VQA score: 77.1 -> 78.7 | Helped |
| Add CLIP scoring to VQA and aesthetic scoring (ablation) | SD v1.5 preference construction · TIFA | VQA score: 77.2 -> 78.7 | Helped |
| Add VQA scoring to CLIP and aesthetic scoring (ablation) | SD v1.5 preference construction · TIFA | aesthetic score: 47.2 -> 49.1 | Helped |
| Add aesthetic scoring to CLIP and VQA scoring (ablation) | SD v1.5 preference construction · TIFA | aesthetic score: 47.0 -> 49.1 | Helped |
| Use CLIP score alone instead of all three scores (ablation) | SD v1.5 preference construction · TIFA | aesthetic score: 49.1 -> 47.3 | Hurt |
Surprising findings
- Training on MJHQ-30K (high-quality Midjourney images) gives the highest aesthetic gain for SD v1.4 and SD v1.5, yet AGFSync gives better VQA improvement.
- CLIP score improvements from training are marginal (about 0.2-0.3 on a 0-100 scale).
Limitations
- Relies on LLM and aesthetic scoring models whose biases propagate into the preference data.
- Random noise injection for diversity may reduce prompt-image consistency.
- Only VQA, CLIP and aesthetic scores are used, which may not capture all quality aspects; larger multimodal LLMs were not used for prompt or QA generation due to cost.
Experimental setup
Datasets: TIFA HPS v2 MJHQ-30K
Benchmarks: TIFA HPS v2
Metrics: VQA score aesthetic score CLIP score win rate
Baselines: DPOK DDPO DreamSync StructureDiffusion SynGen MJHQ-30K+SFT
Builds on: Diffusion-DPO DPO DreamSync TIFA
Closest papers by meaning
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
- DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.88)
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.88)
- Preference Adaptive and Sequential Text-to-Image Generation (ICML 2025 · similarity 0.88)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
- Minority-Focused Text-to-Image Generation via Prompt Optimization (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.