Research map / Image generation / Text-to-image diffusion
PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a shared LLaMA-7B with per-dataset LoRA-style adapters on text-continuation, text-revision and GPT-3.5-generated chain-of-thought data to rewrite user prompts so they match high-quality diffusion training captions.
Problem
Diffusion text-to-image and text-to-video generators produce quality that depends heavily on prompt quality, and manual prompt engineering requires expertise; user prompts often differ in distribution from the high-quality captions seen in training.
Approach
Align user prompts to the distribution of high-quality training captions by fine-tuning an LLM on continuation and revision data, then add a five-step chain-of-thought dataset built with ChatGPT, and share one LLM across datasets via lightweight adapters.
Key results
- With SD v2.1 on COCO captions, PromptCoT raises aesthetic score from 5.60 to 5.89 and CLIP score from 0.266 to 0.295, and cuts FID from 58.02 to 40.47.
- On MagicVideo with MSR-VTT, PromptCoT lowers FID from 36.5 to 33.2 and FVD from 998 to 951 versus the original prompts.
- Adapter-based PromptCoT reaches aesthetic 5.85 and FID 51.06, compared with 5.40 and 59.15 for baseline captions, approaching the fully fine-tuned model reported in the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Refine COCO captions with PromptCoT | Stable Diffusion v1.4 · COCO | Aesthetic Score: 5.40 -> 5.79 | Helped |
| Refine COCO captions with PromptCoT | Stable Diffusion v1.4 · COCO | FID: 59.15 -> 41.10 | Helped |
| Refine COCO captions with PromptCoT | Stable Diffusion v1.4 · COCO | CLIP Score: 0.268 -> 0.294 | Helped |
| Refine COCO captions with PromptCoT | Stable Diffusion v2.1 · COCO | Aesthetic Score: 5.60 -> 5.89 | Helped |
| Refine COCO captions with PromptCoT | Stable Diffusion v2.1 · COCO | FID: 58.02 -> 40.47 | Helped |
| Refine COCO captions with PromptCoT | Stable Diffusion v2.1 · COCO | CLIP Score: 0.266 -> 0.295 | Helped |
| Refine prompts with PromptCoT | MagicVideo · MSR-VTT | FID: 36.5 -> 33.2 | Helped |
| Refine prompts with PromptCoT | MagicVideo · MSR-VTT | FVD: 998 -> 951 | Helped |
| Refine prompts with PromptCoT | MagicVideo · MSR-VTT | CLIP Score: 0.284 -> 0.296 | Helped |
| Refine prompts with PromptCoT | COCO subset (1k images) · COCO subset (1k images) | PickScore: 16.8%/26.1% -> 57.5%/73.6% | Helped |
| Use adapters instead of full fine-tuning (ablation) | PromptCoT with Stable Diffusion v1.4 · COCO | Aesthetic Score: 5.79 -> 5.85 | Helped |
| Use adapters instead of full fine-tuning (ablation) | PromptCoT with Stable Diffusion v1.4 · COCO | FID: 41.10 -> 51.06 | Hurt |
Surprising findings
- Human volunteers' refinements raised CLIP score only modestly (0.270) and PickScore was lower than davinci-003 refinements on some metrics, suggesting automated rewriting can outperform human rewriting on these metrics.
Limitations
- LLMs may generate irrelevant or extraneous information, which the CoT stage is meant to mitigate but evaluation of this is limited
- Evaluation is restricted to COCO captions and a small set of diffusion models (SD v1.4, v2.1, MagicVideo)
- Human comparison uses only 30 volunteers and a 1k-image subset
Experimental setup
Datasets: LAION-aes6plus COCO MSR-VTT WebVid-10M Alpaca
Benchmarks: COCO LAION-aes6plus MSR-VTT
Metrics: FID FVD Inception Score Aesthetic Score CLIP score PickScore
Baselines: t-continue t2t-blip t2t-inter davinci-003 human-refined prompts original COCO captions
Builds on: chain-of-thought prompting LoRA Alpaca instruction tuning Stable Diffusion
Closest papers by meaning
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.91)
- Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models (ICLR 2025 · similarity 0.90)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.90)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.90)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.