Research map / Image generation / Text-to-image diffusion
ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ConceptBed, a 284-concept, ~33K-prompt benchmark and the Concept Confidence Deviation (CCD) metric, which uses oracle classifier confidence to evaluate concept learners (Textual Inversion, DreamBooth, Custom Diffusion) for concept and composition alignment.
Problem
Existing evaluations of personalized text-to-image concept learners use small, hand-picked concept sets and metrics (DINO cosine, KID, CLIPScore) that poorly match human judgments of concept and compositional alignment.
Approach
Build a large concept library (PACS styles, ImageNet objects, CUB attributes) with Visual Genome-derived composite prompts, train oracle classifiers on the concepts, and score generated images by the deviation of oracle confidence from its confidence on real target images (CCD); compositions are scored with ViLT VQA existence questions.
Key results
- CCD correlates strongly with human scores (Pearson -0.9515 on Domain PACS and -0.9892 on Object ImageNet), far above DINO and KID.
- Custom Diffusion is worst on concept alignment (CCD 0.2085 on ImageNet objects) but best on composition (CCD 0.0471 relation, 0.0347 action), revealing a concept-composition trade-off.
- Textual Inversion (SD) achieves the best object concept alignment (CCD 0.0472 on ImageNet objects) and CUB object-level (0.0859).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Stable Diffusion-based rather than LDM-based Textual Inversion encoder | Textual Inversion · ObjectsImageNet | Concept Confidence Deviation: 0.0955 -> 0.0472 | Helped |
| Use the Stable Diffusion-based rather than LDM-based Textual Inversion encoder | Textual Inversion · DomainPACS styles | Concept Confidence Deviation: 0.0478 -> 0.2456 | Hurt |
| Use Custom Diffusion instead of Textual Inversion (SD) | Concept learners · Composition | Concept Confidence Deviation: 0.1090 -> 0.4916 | Helped |
| Use Custom Diffusion instead of Textual Inversion (SD) | Concept learners · ObjectsImageNet concept alignment | Concept Confidence Deviation: 0.0472 -> 0.2085 | Hurt |
| Change the classifier used as the concept-alignment oracle (ablation) | Concept Confidence Deviation evaluation · ObjectsImageNet | Pearson correlation with human scores: -0.9892; -0.9888; -0.9816; -0.9763 | No clear effect |
Surprising findings
- CLIPScore rates fine-tuned baselines higher than the original image-text pairs, which the authors consider inaccurate.
- Textual Inversion (LDM) learns the sketch style far better than DreamBooth and Custom Diffusion, which struggle with it.
Limitations
- Only 284 concepts, which cannot cover the infinite space of concepts.
- Benchmark mainly evaluates Stable Diffusion-based concept learners.
- Fine-tuned oracles do not generalize to unseen concepts; a few-shot classifier alternative is only partially explored.
Experimental setup
Datasets: ImageNet PACS CUB Visual Genome MSCOCO GCC SBU
Benchmarks: ConceptBed
Metrics: Concept Confidence Deviation (CCD) DINO cosine similarity KID CLIPScore VQA accuracy Recall (CCD<=0) Pearson correlation with human score
Baselines: Textual Inversion (LDM) Textual Inversion (SD) DreamBooth Custom Diffusion DINO cosine similarity KID CLIPScore
Builds on: Textual Inversion DreamBooth Custom Diffusion Latent Diffusion Model CLIP
Closest papers by meaning
- ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty (NeurIPS 2024 · similarity 0.91)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.91)
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.90)
- ICE: Intrinsic Concept Extraction from a Single Image via Diffusion Models (CVPR 2025 · similarity 0.89)
- Text-to-Image Generation for Abstract Concepts (AAAI 2024 · similarity 0.89)
- Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.