Research map / Image generation / Text-to-image diffusion
Visual Persona: Foundation Model for Full-Body Human Customization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Visual Persona, a full-body human customization model that trains a body-partitioned transformer encoder-decoder (DINOv2 features) on a VLM-curated paired dataset (Visual Persona-500K) to condition a frozen text-to-image diffusion model.
Problem
Text-to-image customization of full-body human appearance from a single in-the-wild image, where prior methods focus on faces or train on single images per individual, and paired multi-image full-body human data is scarce.
Approach
Curate a paired full-body human dataset from unpaired collections using VLM-based clothing consistency checks, and encode body-part crops with DINOv2 into dense identity embeddings via a body-partitioned transformer decoder that conditions a frozen T2I diffusion model through an added identity cross-attention.
Key results
- On PPR10K, Visual Persona reaches D-I 7.30 and D-H 6.85, versus 6.80/6.63 for StoryMaker and 2.81 for IP-Adapter-FaceID (D-H).
- On SSHQ, Visual Persona achieves D-I 7.10 and D-H 6.99, outperforming StoryMaker (6.74 D-I, 6.71 D-H).
- Human evaluation overall score O is highest for Visual Persona (about 0.79 vs. 0.55 for StoryMaker and 0.31 for IP-Adapter in the figure).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add cross-attention layer (transformer decoder) to MLP-on-DINOv2 projection (ablation) | identity embedding projection · PPR10K | D-I: 6.66 -> 7.47 | Helped |
| Add cross-attention layer (transformer decoder) to MLP-on-DINOv2 projection (ablation) | identity embedding projection · PPR10K | D-T: 7.11 -> 6.13 | Hurt |
| Add body part decomposition to transformer decoder (ablation) | identity embedding projection · PPR10K | D-T: 6.13 -> 6.67 | Helped |
| Add body part decomposition to transformer decoder (ablation) | identity embedding projection · PPR10K | D-H: 6.40 -> 6.85 | Helped |
| Increase identity embedding token length from 4x4 to 16x16 (dense) (ablation) | identity embedding · PPR10K | D-I: 5.51 -> 7.30 | Helped |
| Increase identity embedding token length from 4x4 to 16x16 (dense) (ablation) | identity embedding · PPR10K | D-H: 5.81 -> 6.85 | Helped |
| Swap body parsing model from SSHQ/PPR10K-specific to SCHP (ablation) | Visual Persona · SSHQ | D-H: 6.99 -> 7.10 | No clear effect |
| Swap body parsing model from SSHQ/PPR10K-specific to SCHP (ablation) | Visual Persona · PPR10K | D-H: 6.85 -> 7.00 | No clear effect |
| Use Visual Persona full-body customization instead of IP-Adapter baseline | full-body customization · PPR10K | D-I: 5.04 -> 7.30 | Helped |
| Use Visual Persona instead of StoryMaker | full-body customization · PPR10K | D-H: 6.63 -> 6.85 | Helped |
| Use Visual Persona instead of StoryMaker | full-body customization · PPR10K | D-T: 6.77 -> 6.67 | Hurt |
Surprising findings
- Using an MLP instead of a transformer decoder gives text alignment comparable to the full model, while adding cross-attention boosts identity preservation at the cost of text alignment; body-part decomposition recovers text alignment.
Limitations
- Evaluation relies on GPT-based scoring, which has difficulty detecting facial expressions for distant subjects.
- Training data curation depends on VLM judgments of clothing consistency, which may be imperfect.
Experimental setup
Datasets: Visual Persona-500K SSHQ PPR10K
Benchmarks: SSHQ PPR10K
Metrics: DreamBench++ D-I DreamBench++ D-T DreamBench++ D-H human semantic consistency human perceptual quality
Baselines: IP-Adapter-FaceID InstantID PhotoMaker IP-Adapter StoryMaker
Builds on: IP-Adapter DINOv2 Stable Diffusion SCHP
Closest papers by meaning
- PersonaHOI: Effortlessly Improving Face Personalization in Human-Object Interaction Generation (CVPR 2025 · similarity 0.89)
- PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium (AAAI 2025 · similarity 0.88)
- PersonaBooth: Personalized Text-to-Motion Generation (CVPR 2025 · similarity 0.86)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.86)
- CapHuman: Capture Your Moments in Parallel Universes (CVPR 2024 · similarity 0.86)
- PLIP: Language-Image Pre-training for Person Representation Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.