Research map / Image generation / Editing and style transfer
CapHuman: Capture Your Moments in Parallel Universes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CapHuman, a tuning-free framework built on Stable Diffusion 1.5 that generates identity-preserved human portraits from one reference face with text control and 3DMM-based head control.
Problem
One-shot human-centric image synthesis that preserves a specific individual's identity while allowing text-driven context and fine-grained head pose, expression, and illumination control, without test-time fine-tuning.
Approach
Encodes global (face-recognition) and local (CLIP patch) identity features and aligns them into a trainable CapFace side network via cross-attention; adds 3DMM (FLAME/DECA) rendered normal, albedo and Lambertian conditions for 3D-consistent head control, with time-dependent ID dropout, while keeping Stable Diffusion frozen.
Key results
- CapHuman reaches 0.8363 ID similarity versus 0.6860 for DreamBooth and 0.6191 for FastComposer on HumanIPHC.
- Head control shape RMSE drops to 0.1020 from 0.1542 for the best baseline (DreamBooth), and lighting to 0.0965 from 0.1634 (LoRA).
- Removing 3DMM conditioning raises shape RMSE from 0.1381 to 0.2909 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CapHuman instead of DreamBooth | CapHuman · HumanIPHC | ID sim.: 0.6860 -> 0.8363 | Helped |
| use CapHuman instead of ControlNet | CapHuman · HumanIPHC | CLIP score: 0.2479 -> 0.2256 | Hurt |
| use CapHuman instead of ControlNet | CapHuman · HumanIPHC | Prompt acc.: 90.32% -> 74.17% | Hurt |
| use CapHuman instead of DreamBooth | CapHuman · HumanIPHC | Shape: 0.1542 -> 0.1020 | Helped |
| use CapHuman instead of DreamBooth | CapHuman · HumanIPHC | Pose: 0.0441 -> 0.0436 | Helped |
| use CapHuman instead of DreamBooth | CapHuman · HumanIPHC | Exp.: 0.1922 -> 0.1241 | Helped |
| use CapHuman instead of LoRA | CapHuman · HumanIPHC | Light.: 0.1634 -> 0.0965 | Helped |
| add global and local identity features (ablation) | CapHuman · HumanIPHC 10 identities | ID sim.: 0.3915 -> 0.8429 | Helped |
| add local identity feature (ablation) | CapHuman · HumanIPHC 10 identities | ID sim.: 0.7725 -> 0.8429 | Helped |
| add global identity feature (ablation) | CapHuman · HumanIPHC 10 identities | ID sim.: 0.8095 -> 0.8429 | Helped |
| use 3DMM facial prior (ablation) | CapHuman · HumanIPHC 10 identities | Shape: 0.2909 -> 0.1381 | Helped |
| increase ID dropout start timestep τ (ablation) | CapHuman · HumanIPHC 10 identities | ID sim.: 0.3915 -> 0.8429 | Helped |
Surprising findings
- Fine-tuning baselines lose prompt control under one-shot settings, with prompt accuracy reflecting overfitting.
- ControlNet alone gives very low identity similarity (0.0534), showing pose conditioning does not preserve identity.
Limitations
- Identity preservation trades off against head pose control via the ID dropout start timestep.
- Relies on a single reference image and a 3DMM/DECA reconstruction pipeline.
- Benchmark HumanIPHC is self-constructed and evaluated with automatic metrics only.
Experimental setup
Datasets: CelebA LAION-5B
Benchmarks: HumanIPHC
Metrics: ID similarity (cosine) CLIP score prompt accuracy RMSE of DECA shape/pose/expression/lighting
Baselines: ControlNet Textual Inversion LoRA DreamBooth FastComposer
Builds on: Stable Diffusion ControlNet FLAME DECA CLIP
Closest papers by meaning
- MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans (NeurIPS 2025 · similarity 0.88)
- Synthetic Prior for Few-Shot Drivable Head Avatar Inversion (CVPR 2025 · similarity 0.88)
- HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion (ICLR 2024 · similarity 0.88)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.87)
- GaussianIP: Identity-Preserving Realistic 3D Human Generation via Human-Centric Diffusion Prior (CVPR 2025 · similarity 0.87)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.