Research map / Vision-language and multimodal / Human and embodied modeling

U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation

CVPR 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Language-grounded scene text and structure · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes U-VAP, which personalizes user-specified visual attributes (color, pattern, structure) from few reference images using LLM-generated target/non-target prompts, dual identifier learning on self-augmented data, and embedding-space semantic adjustment at inference.

Problem

Fine-grained visual appearance personalization: extracting a user-specified visual attribute (e.g., color, pattern, material, structure) from a few reference images of a concept, since existing concept-level personalization entangles all attributes together.

Approach

A decoupled self-augmentation strategy: an LLM (GPT-3.5-turbo) writes target-attribute and non-target-attribute prompts, a pre-trained DreamBooth model generates images for each set, CLIP-based filtering curates them, and two identifiers (tgt, ngt) are learned on the separate sets. At inference, the tgt embedding is shifted away from the ngt embedding (semantic adjustment) to reduce entangled non-

Key results

• On color attribute, U-VAP achieves CLIP-T 0.292 and IS 4.138, versus DreamBooth 0.303/3.890 and ProSpect 0.239/2.691.
• On pattern attribute, U-VAP reaches CLIP-T 0.270 and IS 3.434, with DreamBooth at CLIP-T 0.299 and IS 3.793.
• On structure attribute, U-VAP has the highest CLIP-T (0.284) versus DreamBooth 0.265 and ProSpect 0.259, with lower CLIP-I (0.709 vs DreamBooth 0.837).

Measured effects reported in the paper

Surprising findings

• DreamBooth scores higher than U-VAP on CLIP-T for color and pattern attributes, which the authors attribute to DreamBooth over-editing the concept.
• Standard CLIP-T alone cannot reflect editing accuracy because over-editing inflates prompt similarity.

Limitations

• Performance depends on the underlying concept-aware personalization method (e.g., DreamBooth), which can decouple some attributes poorly during pre-learning.
• Data curation relies on CLIP similarity filtering and human feedback selection, which adds manual effort.
• The semantic adjustment strength lambda must be chosen at inference and its optimal value is not fixed.

Experimental setup

Datasets: few-shot reference images of concepts LLM-generated target and non-target text prompts images generated by initial DreamBooth model

Benchmarks: in-house set of 1272 generated images across color, pattern, and structure attributes

Metrics: CLIP-I CLIP-T Inception Score

Baselines: Textual Inversion DreamBooth GPT-4V ProSpect Null-text Inversion + Prompt-to-Prompt Concept Decomposition

Builds on: DreamBooth Stable Diffusion Textual Inversion ProSpect

Closest papers by meaning

• Visual Persona: Foundation Model for Full-Body Human Customization (CVPR 2025 · similarity 0.79)
• High-Quality Facial Geometry and Appearance Capture at Home (CVPR 2024 · similarity 0.79)
• Contextualized Visual Personalization in Vision-Language Models (ICML 2026 · similarity 0.79)
• Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generation (CVPR 2024 · similarity 0.79)
• MagiCapture: High-Resolution Multi-Concept Portrait Customization (AAAI 2024 · similarity 0.78)
• Foundation Cures Personalization: Improving Personalized Models’ Prompt Consistency via Hidden Foundation Knowledge (NeurIPS 2025 · similarity 0.77)