Research map / Image generation / Text-to-image diffusion
MagiCapture: High-Resolution Multi-Concept Portrait Customization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MagiCapture, a few-shot multi-concept personalization method that fuses subject and style references into high-resolution portrait images using masked reconstruction, composed prompt learning with pseudo-labels, and an Attention Refocusing loss.
Problem
Personalizing text-to-image diffusion models to compose a specific subject's identity with a reference style from only a few images, without ground truth for composed concepts, leading to identity shift and low realism in portrait generation.
Approach
Two-phase (embedding then LoRA) optimization with masked reconstruction of facial and non-facial regions, a composed-prompt training objective using reference images as pseudo-labels plus a face-identity loss, and an Attention Refocusing loss that pushes special-token cross-attention to zero outside the target mask.
Key results
- MagiCapture achieves CSIM 0.566 vs 0.436 for Custom Diffusion (best baseline) and aesthetic 6.218 vs 6.178 for the no-AR/CP variant.
- In a 30-person user study, MagiCapture scores 4.055 ID, 4.165 style, 4.293 fidelity, above all baselines (best baseline ID 3.223).
- Ablation shows removing composed prompt learning drops CSIM from 0.566 to 0.429.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add composed prompt learning (ablation) | MagiCapture · Portrait customization | CSIM: 0.429 -> 0.566 | Helped |
| add composed prompt learning (ablation) | MagiCapture · Portrait customization | Style: 0.717 -> 0.730 | Helped |
| add composed prompt learning (ablation) | MagiCapture · Portrait customization | Aesthetic: 6.159 -> 6.218 | Helped |
| add Attention Refocusing loss (ablation) | MagiCapture · Portrait customization | CSIM | No clear effect |
| add Attention Refocusing loss (ablation) | MagiCapture · Portrait customization | information disentanglement | Helped |
| add super-resolution and face restoration postprocessing (ablation) | MagiCapture · Portrait customization | CSIM: 0.566 -> 0.508 | Hurt |
| add super-resolution and face restoration postprocessing (ablation) | MagiCapture · Portrait customization | Style: 0.730 -> 0.737 | Helped |
| add super-resolution and face restoration postprocessing (ablation) | MagiCapture · Portrait customization | Aesthetic: 6.218 -> 6.184 | Hurt |
| use MagiCapture instead of DreamBooth | Stable Diffusion V1.5 · Portrait customization | CSIM: 0.102 -> 0.566 | Helped |
| use MagiCapture instead of Textual Inversion | Stable Diffusion V1.5 · Portrait customization | CSIM: 0.224 -> 0.566 | Helped |
| use MagiCapture instead of Custom Diffusion | Stable Diffusion V1.5 · Portrait customization | CSIM: 0.436 -> 0.566 | Helped |
Surprising findings
- Attention Refocusing loss does not improve automatic metrics, yet its absence causes collapsed samples and information spill into earrings.
- Post-processing with face restoration lowers CSIM from 0.566 to 0.508 while slightly improving style score.
Limitations
- Occasionally produces abnormal body parts such as limbs and fingers
- Lower fidelity for non-white subjects and gender bias inherited from the pretrained model
- Post-processing face restoration can slightly reduce identity similarity
Experimental setup
Datasets: VGGFace in-the-wild internet identity photos
Benchmarks: VGGFace identities with curated style set (100 id-style pairs)
Metrics: CSIM masked CLIP style similarity LAION aesthetic score user study ID/style/fidelity scores
Baselines: DreamBooth Textual Inversion Custom Diffusion
Builds on: DreamBooth Pivotal Tuning Stable Diffusion LoRA Custom Diffusion
Closest papers by meaning
- PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization (CVPR 2024 · similarity 0.87)
- MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion (ICML 2024 · similarity 0.87)
- DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation (CVPR 2024 · similarity 0.86)
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.86)
- EchoShot: Multi-Shot Portrait Video Generation (NeurIPS 2025 · similarity 0.86)
- PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.