Research map / Image generation / Editing and style transfer
SHMT: Self-supervised Hierarchical Makeup Transfer via Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SHMT, a self-supervised latent-diffusion makeup transfer method that decouples content (3D shape plus Laplacian-pyramid texture) from makeup and uses an Iterative Dual Alignment module, avoiding pseudo-paired supervision.
Problem
Makeup transfer lacks paired ground truth, so prior methods train on sub-optimal pseudo-paired data (from histogram matching or geometric warping), yielding low makeup fidelity; they also cannot flexibly preserve or discard source texture details across makeup styles of varying complexity.
Approach
Self-supervised "decoupling-and-reconstruction": the model reconstructs the source face from its own content representation (3DDFA-V2 face shape plus a Laplacian-pyramid texture level) and a spatially distorted makeup representation, with the choice of texture level controlling how much source detail is kept. An IDA module mixes a reference-based alignment with an alignment computed from the noisy
Key results
- On MT, SHMT-h4 reaches FID 24.93 and CLS 0.715 versus best baseline Stable-Makeup at 33.26 and 0.682 (FID lower is better).
- SHMT-h0 gets the highest Key-sim (0.976 on MT) for content preservation, indicating a fidelity–preservation tradeoff controlled by the texture level.
- Removing IDA raises FID from 27.01 to 32.42 and lowers CLS from 0.786 to 0.753 on LADN for SHMT-h4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IDA's additional alignment prediction to the injection condition (ablation) | SHMT-h4 · LADN | FID: 32.42 -> 27.01 | Helped |
| Add IDA's additional alignment prediction to the injection condition (ablation) | SHMT-h4 · LADN | CLS: 0.753 -> 0.786 | Helped |
| Add IDA's additional alignment prediction to the injection condition (ablation) | SHMT-h4 · LADN | Key-sim: 0.960 -> 0.958 | Hurt |
| Use progressively coarser texture detail, from h0 to h4 (ablation) | SHMT · LADN | FID: 38.97 -> 38.86 -> 36.35 -> 33.12 -> 27.01 | Helped |
| Use progressively coarser texture detail, from h0 to h4 (ablation) | SHMT · LADN | CLS: 0.711 -> 0.712 -> 0.726 -> 0.748 -> 0.786 | Helped |
| Use progressively coarser texture detail, from h0 to h4 (ablation) | SHMT · LADN | Key-sim: 0.978 -> 0.978 -> 0.973 -> 0.965 -> 0.958 | Hurt |
Surprising findings
- The paper reports a tradeoff: the variant with the most texture detail preserved has the best content similarity but the worst makeup fidelity and FID.
Limitations
- Depends on the accuracy of pretrained face parsing and 3D face reconstruction models; errors degrade results (e.g., forehead makeup marked as hair).
- Higher parameter count and inference cost than GAN-based methods, several seconds per image.
- Makeup fidelity and content preservation trade off against each other via texture-level choice.
Experimental setup
Datasets: MT Wild-MT LADN
Benchmarks: MT Wild-MT LADN
Metrics: FID CLS similarity Key-sim user study preference
Baselines: PSGAN SCGAN EleGANt SSAT LADN CPM Stable-Makeup
Builds on: Latent Diffusion Models DDIM 3DDFA-V2 Laplacian pyramid
Closest papers by meaning
- Content-Style Decoupling for Unsupervised Makeup Transfer without Generating Pseudo Ground Truth (CVPR 2024 · similarity 0.89)
- Makeup Prior Models for 3D Facial Makeup Estimation and Applications (CVPR 2024 · similarity 0.87)
- Self-Supervised Selective-Guided Diffusion Model for Old-Photo Face Restoration (NeurIPS 2025 · similarity 0.86)
- Multi-focal Conditioned Latent Diffusion for Person Image Synthesis (CVPR 2025 · similarity 0.86)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.85)
- Diffusion Prior Interpolation for Flexibility Real-World Face Super-Resolution (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.