Research map / Image generation / Text-to-image diffusion
An Image is Worth Multiple Words: Discovering Object Level Concepts using Multi-Concept Prompt Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-Concept Prompt Learning (MCPL), which learns multiple object-level word embeddings from one image-sentence pair without masks or crops, using attention masking, a prompt contrastive loss, and adjective binding on a frozen Textual Inversion/LDM setup.
Problem
Learning multiple unknown object-level concepts from a single image and its sentence without image annotations (masks or crops), so that concepts can be discovered and edited locally from text alone.
Approach
Learn one textual embedding per noun in a descriptive sentence from a single unannotated image, keeping the diffusion model frozen, and steer the embeddings with three regularisers: cross-attention-derived masks restricting the loss (AttnMask), an InfoNCE-style contrastive loss separating concept embeddings (PromptCL), and binding each noun to an adjective as a positive group (Bind adj.).
Key results
- Fully regularised MCPL-one is competitive with the mask-based Break-A-Scene on the natural dataset in embedding similarity, while BAS leads on DINOv1 for the medical dataset.
- Learned token storage is under 0.1 MB versus 4.9 GB for Break-A-Scene, since only text embeddings are updated.
- In a 41-person user study, MCPL-one with AttnMask, PromptCL and Bind adj. was preferred in 40.51% of text alignment and 65.64% of medical semantic correspondence quizzes, the highest among variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MCPL-one with PromptCL and AttnMask (ablation) | MCPL · User study: text alignment | percentage of users: 15.38 -> 40.51 | Helped |
| Use MCPL-one with PromptCL and AttnMask (ablation) | MCPL · User study: prompt following | percentage of users: 21.28 -> 56.15 | Helped |
| Use MCPL-one with PromptCL and AttnMask (ablation) | MCPL · User study: semantic correspondence (natural images) | percentage of users: 26.67 -> 54.36 | Helped |
| Use MCPL-one with PromptCL and AttnMask (ablation) | MCPL · User study: semantic correspondence (medical images) | percentage of users: 13.33 -> 65.64 | Helped |
| Add PromptCL to MCPL-all (ablation) | MCPL-all · User study: text alignment | percentage of users: 7.18 -> 7.95 | Helped |
| Add PromptCL to MCPL-all (ablation) | MCPL-all · User study: prompt following | percentage of users: 3.08 -> 2.31 | Hurt |
| Add PromptCL to MCPL-one (ablation) | MCPL-one · User study: semantic correspondence (natural images) | percentage of users: 7.18 -> 26.67 | Helped |
| Add PromptCL to MCPL-one (ablation) | MCPL-one · User study: semantic correspondence (medical images) | percentage of users: 5.13 -> 13.33 | Helped |
| Use MCPL-diverse rather than MCPL-one for per-image varying concepts and relationships (ablation) | MCPL · Per-image different concept tasks | qualitative performance | Helped |
| Include adjective words rather than remove them (ablation) | MCPL-diverse · Prompt-region correlation | qualitative performance | Helped |
| Include all proposed regularisation terms together rather than applying them in isolation (ablation) | MCPL · Prompt-concept correlation | qualitative performance | Helped |
| Learn only noun prompts rather than all prompts in the string (ablation) | MCPL · Prompt-concept correlation | qualitative performance | Helped |
Surprising findings
- Plain multi-concept textual inversion with masked or cropped single-concept images yields distinct embeddings, but composing them into one scene remains difficult even when each concept is well learned.
- Adjective words often correlate strongly with specific image regions in the pretrained model, and binding them to nouns improves object-level correlation.
Limitations
- Performance declines as the number of concepts per image grows, and learning from a single image limits accuracy.
- Attention-derived masks are less accurate than segmentation models such as BAS's human-in-the-loop masks, especially on medical images.
- Struggles when concepts are linguistically indistinct, such as multiple identical or highly similar instances.
Experimental setup
Datasets: MaskFormer training data COCO and ADE20K (via pretrained segmenter) Unsplash real images P2P image examples LGE-CMR cardiac MRI MIMIC-CXR chest X-rays
Benchmarks: MCPL multi-concept dataset (natural and biomedical) Unsplash real image dataset
Metrics: t-SNE clustering of learned embeddings prompt fidelity (BERT cosine similarity) image fidelity (CLIP, DINOv1, DINOv2 cosine similarity) user study preference percentage
Baselines: Textual Inversion (masked) Break-A-Scene MCPL-all Custom Diffusion Cones
Builds on: Textual Inversion Latent Diffusion Models Break-A-Scene Prompt-to-Prompt
Closest papers by meaning
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
- Decoupled Textual Embeddings for Customized Image Generation (AAAI 2024 · similarity 0.88)
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.88)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.87)
- Concept-Guided Prompt Learning for Generalization in Vision-Language Models (AAAI 2024 · similarity 0.86)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.