Research map / Large language models / Language, safety and interpretability
DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffAgent, a LLaMA-2-7B agent that selects Stable Diffusion models and parameters (T2I APIs) for a user prompt, trained with SFT then RRHF alignment using an image-score reward on a new Civitai-derived dataset, DABench.
Problem
Selecting the right personalized Stable Diffusion model and generation parameters from tens of thousands of community models requires many manual trials; the task is to pick a suitable T2I API for a user prompt quickly.
Approach
Treat T2I model selection as LLM API calling: fine-tune an LLM on Civitai-derived prompt-API pairs, then align it with human preferences by extending RRHF so that the ranking reward is a unified image-quality score (CLIP score, ImageReward, HPS v2) computed on generated images from each candidate API.
Key results
- DiffAgent-RRHF improves the Unified Metric over the baseline by 18.8 to 31 points on SD 1.5 and 8.3 to 9.4 on SDXL.
- RRHF reduces hallucination error on Parti Prompts for SD 1.5 from 28.9% (SFT) to 20.0% and on SDXL from 48.1% to 28.0%.
- In a 10-user study, DiffAgent won 78% on relevance and 86% on human preference versus the baseline on SD 1.5; average API generation time is 4.81s on a single A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RRHF alignment after SFT (ablation) | DiffAgent · DABench, COCO Caption, and Parti Prompts (SD XL) | Unified Metric: 55.8 -> 57.5; 55.6 -> 57.4; 56.5 -> 58.8 | Helped |
| Add RRHF alignment after SFT (ablation) | DiffAgent · DABench, COCO Caption, and Parti Prompts (SD 1.5) | Unified Metric: 59.7 -> 68.2; 53.5 -> 62.2; 53.7 -> 61.8 | Helped |
| Use DiffAgent-RRHF parameter information with the baseline model (ablation) | Baseline · DABench, COCO Caption, and Parti Prompts (SD XL) | Unified Metric: 49.2 -> 54.2; 48.0 -> 53.7; 49.8 -> 53.6 | Helped |
| Use DiffAgent-RRHF model information with default parameters (ablation) | Baseline · DABench, COCO Caption, and Parti Prompts (SD XL) | Unified Metric: 49.2 -> 53.0; 48.0 -> 52.2; 49.8 -> 52.5 | Helped |
| Use DiffAgent-RRHF parameter information with the baseline model (ablation) | Baseline · DABench, COCO Caption, and Parti Prompts (SD 1.5) | Unified Metric: 37.2 -> 39.5; 42.3 -> 45.2; 43.0 -> 46.5 | Helped |
| Use DiffAgent-RRHF model information with default parameters (ablation) | Baseline · DABench, COCO Caption, and Parti Prompts (SD 1.5) | Unified Metric: 37.2 -> 67.0; 42.3 -> 60.8; 43.0 -> 60.7 | Helped |
| Add RRHF alignment after SFT (ablation) | DiffAgent · DABench (SD 1.5) | hallucination error: 11.5 -> 9.5 | Helped |
| Add RRHF alignment after SFT (ablation) | DiffAgent · COCO Caption (SD 1.5) | hallucination error: 26.9 -> 15.5 | Helped |
| Add RRHF alignment after SFT (ablation) | DiffAgent · Parti Prompts (SD 1.5) | hallucination error: 28.9 -> 20.0 | Helped |
| Add RRHF alignment after SFT (ablation) | DiffAgent · DABench (SD XL) | hallucination error: 7.4 -> 7.1 | Helped |
| Add RRHF alignment after SFT (ablation) | DiffAgent · COCO Caption (SD XL) | hallucination error: 45.6 -> 36.0 | Helped |
| Add RRHF alignment after SFT (ablation) | DiffAgent · Parti Prompts (SD XL) | hallucination error: 48.1% -> 28.0% | Helped |
Surprising findings
- Applying the default SD baseline with DiffAgent's parameters only (Baseline*) or model only (DiffAgent#) yields different gains per architecture: model information dominates for SD 1.5, parameter information for SDXL.
Limitations
- Hallucinated APIs remain (SDXL Parti Prompts error still 28.0% after RRHF).
- Reward relies on automatic image metrics (CLIP, ImageReward, HPS v2) and a small user study with 10 users.
- Dataset is restricted to SD 1.5 and SDXL models from Civitai (LoRA and checkpoints).
Experimental setup
Datasets: DABench COCO Caption Parti Prompts
Benchmarks: DABench COCO Caption Parti Prompts
Metrics: CLIP Score ImageReward HPS v2 Unified Metric hallucination error user win rate
Baselines: Stable Diffusion with default parameters DiffAgent-SFT Baseline*
Builds on: Gorilla GPT4Tools RRHF LLaMA-2 ImageReward
Closest papers by meaning
- EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist Rewards (NeurIPS 2026 · similarity 0.86)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.86)
- DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.86)
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models (CVPR 2025 · similarity 0.85)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.