Research map / Image generation / Text-to-image diffusion
Finetuning LLMs for Automatic Concept to TTI Prompt Generation (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Finetunes GPT-2 XL on LAION-400M captions (with bos/eos tags) to generate Stable Diffusion prompts from concept noun phrases, evaluated by a small human survey of image quality and faithfulness.
Problem
Users of text-to-image models need effective prompts to get high-quality images; the paper asks whether an LLM finetuned on TTI training captions can automatically turn a plain subject into higher-quality prompts without human intervention.
Approach
Finetune the upper layers of GPT-2 XL on LAION caption data (with explicit start/end tags marking complete captions) so its generated prompts align with the distribution Stable Diffusion was trained on, improving image quality at some cost to concept faithfulness.
Key results
- Finetuned GPT-2 XL with tags reached image quality 3.04 vs 2.81 for the untuned baseline (No FT).
- Faithfulness dropped from 3.02 (No FT) to 2.77 (FT w/ Tags) and 2.72 (FT No Tags).
- Inter-annotator agreement was weak for image quality (weighted κ = 0.386) and moderate for faithfulness (κ = 0.546).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune GPT2 on captions without BOS/EOS tags | GPT2 · 10 LAION concepts | image quality (Likert rating): +0.15 points | Helped |
| Fine-tune GPT2 on captions without BOS/EOS tags | GPT2 · 10 LAION concepts | faithfulness (Likert rating): −0.30 points | Hurt |
| Add BOS/EOS tags to captions during fine-tuning (ablation) | GPT2 fine-tuned without tags · 10 LAION concepts | image quality (Likert rating): 2.96 -> 3.04 | Helped |
| Add BOS/EOS tags to captions during fine-tuning (ablation) | GPT2 fine-tuned without tags · 10 LAION concepts | faithfulness (Likert rating): 2.72 -> 2.77 | Helped |
Surprising findings
- Finetuning on only about 0.1% of the available captions improved image quality, at the cost of concept faithfulness.
Limitations
- Small human survey with 5 participants from the same lab; weak image-quality agreement (κ = 0.386).
- Finetuning reduced concept faithfulness, with prompts drifting toward stylistic patterns.
- Only 10 concepts and 150 images evaluated.
Experimental setup
Datasets: LAION-400M LAION captions (300,000 captions for concept extraction)
Metrics: image quality Likert score (1-4) concept faithfulness Likert score (1-4) weighted Cohen's kappa
Baselines: GPT-2 XL without finetuning GPT-2 XL finetuned without tags
Builds on: GPT-2 Stable Diffusion LAION-400M
Closest papers by meaning
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.88)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.88)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.88)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.88)
- FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition (CVPR 2024 · similarity 0.87)
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.