Research map / Image generation / Text-to-image diffusion
DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DisEnvisioner, a tuning-free, single-image customized image generator that disentangles subject-essential from irrelevant visual tokens (DisVisioner) and enriches the subject token into granular tokens (EnVisioner) on top of Stable Diffusion v1.5.
Problem
Customized image generation from a single visual prompt struggles to separate subject-essential attributes from subject-irrelevant ones (pose, background, tone), hurting both editability and ID consistency.
Approach
Disentangle image features into one subject token and one irrelevant token via a spatial-softmax cross-attention image tokenizer initialized with CLIP priors, then enrich the subject token into multiple tokens with separate cross-attention injection; at inference the irrelevant branch is switched off (lambda_i=0).
Key results
- DisEnvisioner achieves the highest text alignment C-T of 0.315 and lowest internal variance IV of 0.026 among compared methods in Table 1.
- Inference takes 1.96s per customized generation without test-time tuning, versus 1120s for DreamBooth and 2420s for DisenBooth.
- Best mRank of 2.0 in the automatic evaluation and 1.0 in the 69-user study; IP-Adapter has higher C-I (0.883) and D-I (0.912) than DisEnvisioner.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DisEnvisioner instead of DisenBooth | customized image generation · DreamBooth | C-T: 0.303 -> 0.315 | Helped |
| use DisEnvisioner instead of IP-Adapter | customized image generation · DreamBooth | C-I: 0.883 -> 0.828 | Hurt |
| use DisEnvisioner instead of IP-Adapter | customized image generation · DreamBooth | D-I: 0.912 -> 0.802 | Hurt |
| use DisEnvisioner instead of BLIP-Diffusion | customized image generation · DreamBooth | IV: 0.029 -> 0.026 | Helped |
| use DisEnvisioner instead of BLIP-Diffusion | customized image generation · DreamBooth | inference-time (s): 1.10 -> 1.96 | Hurt |
| use two-stage training rather than combining DisVisioner and EnVisioner training (ablation) | DisEnvisioner · DreamBooth | image customization quality | Helped |
| add EnVisioner enrichment to the disentangled subject representation (ablation) | DisEnvisioner · DreamBooth | ID consistency | Helped |
| use one subject-essential token and one irrelevant token rather than excessive token counts (ablation) | DisVisioner · DreamBooth | image customization quality | Helped |
| increase the subject-essential attention weight λ_s (ablation) | DisEnvisioner · DreamBooth | ID consistency | Helped |
| decrease the subject-irrelevant attention weight λ_i (ablation) | DisEnvisioner · DreamBooth | subject-irrelevant disturbances | Helped |
| provide a class name in the prompt (ablation) | DisEnvisioner · DreamBooth | ID consistency | No clear effect |
Surprising findings
- Single-stage joint training of disentanglement and enrichment failed to capture any subject information, requiring a two-stage strategy.
- IP-Adapter and DreamBooth get higher image-alignment scores but poor editability because they replicate large parts of the reference image.
Limitations
- Relies on a single subject token and a single irrelevant token; more tokens degrade disentanglement.
- Automatic image-alignment metrics favor methods that copy the reference, and DisEnvisioner trails IP-Adapter on C-I and D-I.
Experimental setup
Datasets: OpenImages V6 DreamBooth
Benchmarks: DreamBooth
Metrics: CLIP text alignment (C-T) CLIP image alignment (C-I) DINO image alignment (D-I) Internal Variance (IV) inference time mean rank (mRank)
Baselines: DreamBooth DisenBooth ELITE IP-Adapter BLIP-Diffusion
Builds on: Stable Diffusion BLIP-Diffusion ELITE DreamBooth IP-Adapter
Closest papers by meaning
- Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting (CVPR 2024 · similarity 0.87)
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.87)
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.87)
- Imagine Before You Draw: Visual Prompt Engineering for Image Generation (NeurIPS 2026 · similarity 0.87)
- Attention Calibration for Disentangled Text-to-Image Personalization (CVPR 2024 · similarity 0.87)
- DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.