Research map / Image generation / Text-to-image diffusion
Lego: Learning to Disentangle and Invert Personalized Concepts Beyond Object Appearance in Text-to-Image Diffusion Models
Saman Motamed, Danda Pani Paudel, Luc Van Gool
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| invert concept with Lego instead of describing it in natural language | LDM · Concept preference user study (7 non-numeric concepts) | human preference: preferred over 70% of cases | Improved |
| invert cardinality concept 4 with Lego rather than natural language prompts | LDM · Cardinality concept 4 | images with correct count: 18 -> 63 | Improved |
| invert walking on a rope with Lego rather than natural language | LDM · Walking on a rope (VQA) | images confirmed by Flan-T5 XL: 4 -> 133 | Improved |
| add Subject Separation (explicit subject embedding learned from concept-free images) (ablation) | Lego / Textual Inversion · burnt and melted, closed eyes, walking on rope | qualitative concept fidelity | Improved |
| add contrastive Context Loss with synonym and antonym words (ablation) | Lego · burnt and melted, closed eyes, walking on rope | qualitative concept fidelity | Improved |
| add Context Loss without Subject Separation (ReVersion-like) (ablation) | Textual Inversion · subject-entangled concepts | qualitative concept fidelity | No clear effect |
| fine-tune diffusion model parts (Custom-Diffusion) to invert entangled concepts | Custom-Diffusion · melting, closed eyes | appearance leakage | Hurt |
| increase concept images from 2 to 4 (ablation) | Lego · closed eyes (VQA accuracy) | accuracy: 75% -> 83% | Improved |
| increase concept images from 4 to 8 (ablation) | Lego · closed eyes (VQA accuracy) | accuracy: 83% -> 83% | No clear effect |
| use Lego-learned subject embedding in place of Textual Inversion for subject synthesis | Lego · subject synthesis user study | preference rate: 55% -> 45% | No clear effect |
Closest papers by meaning
- An Image is Worth Multiple Words: Discovering Object Level Concepts using Multi-Concept Prompt Learning (ICML 2024 · similarity 0.89)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.88)
- Decoupled Textual Embeddings for Customized Image Generation (AAAI 2024 · similarity 0.88)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.88)
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
