Research map / Vision-language and multimodal / Human and embodied modeling
Learning Vision and Language Concepts for Controllable Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a nonparametric component-wise identifiability theory for atomic text and vision concepts and a T2I model (ConceptAligner) built on SANA that learns sparse text-to-visual concept connections for controllable generation and editing.
Problem
Learning identifiable atomic multimodal (text and image) concepts and their sparse interactions, so that T2I generation and editing can be controlled at the level of individual concepts without unintended changes.
Approach
Formulates concept learning as latent variable identification with a graphical model linking discrete textual concepts to continuous visual concepts; proves component-wise identifiability under nonparametric conditions that allow both discrete and continuous modalities, then builds a T2I model that conditions a diffusion transformer on learned visual concepts with a sparsity mask and KL regulariza
Key results
- On EMU-Edit, ConceptAligner reaches CLIP-I 0.903, LPIPS 0.357 and DINO 0.835, versus best baseline SANA-Finetune DINO 0.750 and SD3.5-M CLIP-I 0.862.
- On PIE-Bench, ConceptAligner gets CLIP-I 0.879 and DINO 0.651, above SANA-Finetune (0.840 and 0.603).
- In the disentanglement test, subject consistency is 0.946 and prompt consistency 0.814, compared with 0.934 and 0.731 for SANA-Finetune; sampling takes 0.48 s/image versus 0.58 s for SANA on H100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ConceptAligner instead of SANA | text-to-image editing · Emu-Edit | CLIP-I: 0.870 -> 0.903 | Helped |
| Use ConceptAligner instead of SANA | text-to-image editing · Emu-Edit | LPIPS: 0.438 -> 0.357 | Helped |
| Use ConceptAligner instead of SANA | text-to-image editing · Emu-Edit | DINO: 0.741 -> 0.835 | Helped |
| Use ConceptAligner instead of SANA-Finetune | animal action edits · animal identity preservation | Subject consistency: 0.934 -> 0.946 | Helped |
| Use ConceptAligner instead of SANA-Finetune | animal action edits · animal action edits | Prompt consistency: 0.731 -> 0.814 | Helped |
| Add sparsity regularization (ablation) | ConceptAligner · Emu-Edit | CLIP-I: 0.863 -> 0.903 | Helped |
| Add diffusion loss (ablation) | ConceptAligner · Emu-Edit | CLIP-T: 0.171 -> 0.314 | Helped |
| Add KL divergence loss (ablation) | ConceptAligner · Emu-Edit | CLIP-T: 0.206 -> 0.314 | Helped |
Surprising findings
- Removing the diffusion loss gives near-identical image pairs, and removing the KL loss lets the model ignore the text condition.
- Without sparsity regularization the model makes unnecessary changes, e.g., when changing horse to zebra.
Limitations
- Visual concept identification requires captions that are sufficiently informative and diverse; poor caption quality may violate this.
- Identifiability conditions (sufficient variability, sparsity) are assumed and not verified empirically on real data.
- Evaluation relies on automatic metrics and an MLLM judge, with no human evaluation reported.
Experimental setup
Datasets: LAION prompts (via FLUX-S generated images) EMU-Edit PIE-Bench
Benchmarks: EMU-Edit PIE-Bench Disentanglement protocol (ChatGPT-generated animal/action prompts)
Metrics: CLIP-I DINO LPIPS CLIP-T subject consistency prompt consistency
Baselines: SD3.5-M SD3.5-L FLUX-S FLUX-D SANA SANA-Finetune DALLE3
Builds on: SANA Kong et al. (2024) discrete hierarchical model identification Kivva et al. (2021) mixture oracle Sun et al. (2024) multimodal sparse identifiability Morioka & Hyvarinen (2024)
Closest papers by meaning
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
- ICE: Intrinsic Concept Extraction from a Single Image via Diffusion Models (CVPR 2025 · similarity 0.88)
- ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty (NeurIPS 2024 · similarity 0.88)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.87)
- Non-confusing Generation of Customized Concepts in Diffusion Models (ICML 2024 · similarity 0.86)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.