Research map / Image generation / Text-to-image diffusion
Text-to-Image Generation for Abstract Concepts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TIAC, a training-free LLM-driven pipeline that clarifies abstract concepts via WordNet, maps them to concrete objects, retrieves form patterns from user prompts, and composes T2I prompts, plus a new "concept score" metric.
Problem
Text-to-image models are trained mainly on concrete objects and fail to visualize abstract concepts (e.g., rendering distorted letters for words like "fairness"), and existing image metrics (IS, VS) do not capture abstract-concept alignment.
Approach
Decompose abstract concepts into three layers inspired by the three-layer artwork theory (intent via WordNet definitions, object via LLM transformation to concrete objects, form via class-level patterns extracted from user-rated prompts in the SAC dataset), then let an LLM integrate them with few-shot examples into a T2I prompt; the proposed concept score multiplies CLIP visual-semantic similarity
Key results
- On the small-scale dataset, LLM+PE achieved the highest human evaluation score (3.80 vs 1.86 for raw words W) and concept score (1.50).
- On the large-scale dataset (3,400 concepts), the average concept score rose from 0.89 (W) to 1.42 (LLM+PE).
- Inception score and VS similarity did not favor the proposed prompts, motivating the new concept score metric.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM+PE instead of directly prompting with the concept name | Stable Diffusion v2 · Small-scale dataset | HE: 1.86 -> 3.80 | Helped |
| Use LLM+PE instead of directly prompting with the concept name | Stable Diffusion v2 · Small-scale dataset | CS: 0.91 -> 1.50 | Helped |
| Use LLM+PE instead of directly prompting with the concept name | Stable Diffusion v2 · Small-scale dataset | IS: 2.58 -> 2.76 | Helped |
| Use LLM+PE instead of directly prompting with the concept name | Stable Diffusion v2 · Small-scale dataset | VS: 0.24 -> 0.25 | Helped |
| Add transformed objects and extracted form patterns to LLM prompt generation (ablation) | LLM prompts for Stable Diffusion v2 · Small-scale dataset | HE: 3.34 -> 3.52 | Helped |
| Add transformed objects and extracted form patterns to LLM prompt generation (ablation) | LLM prompts for Stable Diffusion v2 · Small-scale dataset | CS: 1.14 -> 1.32 | Helped |
| Add transformed objects and extracted form patterns to LLM prompt generation (ablation) | LLM prompts for Stable Diffusion v2 · Small-scale dataset | IS: 2.75 -> 2.62 | Hurt |
| Add transformed objects and extracted form patterns to LLM prompt generation (ablation) | LLM prompts for Stable Diffusion v2 · Small-scale dataset | VS: 0.27 -> 0.25 | Hurt |
| Add eight few-shot prompt examples (ablation) | LLM+P prompts for Stable Diffusion v2 · Small-scale dataset | HE: 3.52 -> 3.80 | Helped |
| Add eight few-shot prompt examples (ablation) | LLM+P prompts for Stable Diffusion v2 · Small-scale dataset | CS: 1.32 -> 1.50 | Helped |
| Add eight few-shot prompt examples (ablation) | LLM+P prompts for Stable Diffusion v2 · Small-scale dataset | IS: 2.62 -> 2.76 | Helped |
| Add eight few-shot prompt examples (ablation) | LLM+P prompts for Stable Diffusion v2 · Small-scale dataset | VS: 0.25 -> 0.25 | No clear effect |
Surprising findings
- Standard image metrics IS and VS did not agree with human ratings: LLM+P and LLM+PE were not superior on IS and VS, with VS slightly favoring the raw-word baseline W+D.
- Directly feeding abstract concept names into T2I models often yields images with distorted English letters instead of meaningful visuals.
Limitations
- Assumes the input concept maps precisely to a WordNet node; real inputs need extra disambiguation effort.
- Human evaluation uses only three participants and a small 57-concept set.
- Concept score relies on an aesthetic-score coefficient and CLIP similarity, which are designed for concrete objects and not fully validated.
Experimental setup
Datasets: Simulacra Aesthetic Captions (SAC) WordNet abstract subtree
Benchmarks: Small-scale WordNet abstract concept set (57 concepts) Large-scale WordNet abstract concept set (3,400 concepts)
Metrics: human evaluation (1-5 rating) concept score inception score visual-semantic similarity (CLIP)
Baselines: W (raw concept words) W+D (words plus definition) LLM (LLM prompt from name and definition) LLM+P (LLM with object and form patterns)
Builds on: Stable Diffusion 2 GPT-3.5 WordNet Simulacra Aesthetic Captions
Closest papers by meaning
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.89)
- ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty (NeurIPS 2024 · similarity 0.89)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.89)
- A Vision Check-up for Language Models (CVPR 2024 · similarity 0.87)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.87)
- ScImage: How good are multimodal large language models at scientific text-to-image generation? (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.