Research map / Image generation / Editing and style transfer
FuseAnyPart: Diffusion-Driven Facial Parts Swapping via Multiple Reference Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FuseAnyPart, a diffusion-based method that swaps multiple facial parts (eyes, nose, mouth) from multiple reference faces by fusing mask-based part features in latent space and injecting them into the UNet via addition.
Problem
Facial parts swapping: transferring selected facial regions from multiple source faces onto a target face while keeping the rest unchanged, which existing full-face swapping methods handle poorly or inefficiently.
Approach
A Mask-based Fusion Module assembles region features from several faces into one complete face feature map in latent space, and an Addition-based Injection Module adds this map into UNet cross-attention outputs instead of using cross-attention adapters.
Key results
- On FF++, FuseAnyPart achieves FID 10.54, the best among compared methods (Diffswap 12.07, StableDiffusion 18.57).
- Addition-based injection in cross-attention layers improves FPSim-E from 0.2542 (cross-attention only) to 0.3186 in the ablation.
- On same-ID reconstruction, DiffSwap reaches MSE 0.15 versus 0.77 for FuseAnyPart, which the authors attribute to DiffSwap copying the source face.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FuseAnyPart instead of Diffswap | FaceForensics++ facial-parts swapping · FaceForensics++ | FID: 12.07 -> 10.54 | Helped |
| use FuseAnyPart instead of FacePartsSwap | FaceForensics++ facial-parts swapping · FaceForensics++ | FPSim-E: 0.3269 -> 0.3186 | No clear effect |
| use FuseAnyPart instead of E4S | FaceForensics++ facial-parts swapping · FaceForensics++ | FPSim-N: 0.4047 -> 0.2234 | Hurt |
| use FuseAnyPart instead of FacePartsSwap | FaceForensics++ facial-parts swapping · FaceForensics++ | FPSim-M: 0.2220 -> 0.2196 | Hurt |
| use FuseAnyPart instead of Diffswap | same-ID face reconstruction · FaceForensics++ | MSE: 0.15 -> 0.77 | Hurt |
| replace addition-based injection with cross-attention (ablation) | FuseAnyPart · FaceForensics++ feature-fusion ablation | FID: 10.54 -> 15.81 | Hurt |
| use multiple separate cross-attention modules for facial parts (ablation) | FuseAnyPart · FaceForensics++ feature-fusion ablation | FID: 10.54 -> 15.32 | Hurt |
| add conditional features in convolutional layers rather than only cross-attention layers (ablation) | FuseAnyPart · FaceForensics++ feature-fusion ablation | FID: 10.54 -> 16.66 | Hurt |
| combine cross-attention with addition in cross-attention layers (ablation) | FuseAnyPart · FaceForensics++ feature-fusion ablation | FPSim-E: 0.3186 -> 0.3108 | Mixed |
| add hierarchical CLIP features to cross-attention plus addition in cross-attention layers (ablation) | FuseAnyPart · FaceForensics++ feature-fusion ablation | FID: 10.51 -> 28.96 | Hurt |
Surprising findings
- Adding multiple cross-attention adapters per facial part gave only limited improvement over a single cross-attention adapter.
- Using hierarchical CLIP features sharply degraded FID (10.51 to 28.96).
Limitations
- Struggles with extreme poses, occlusions, and expressions.
- Relies on high-quality training data; skin color changes can occur in generated images.
- Diffusion inference is computationally heavy; speedups (LCM, int8) are discussed but not evaluated.
Experimental setup
Datasets: CelebA-HQ FaceForensics++
Benchmarks: FaceForensics++
Metrics: FID FPSim-E FPSim-N FPSim-M MSE
Baselines: Stable Diffusion IP-Adapter FacePartsSwap E4S DiffSwap DiffFace
Builds on: Stable Diffusion IP-Adapter CLIP
Closest papers by meaning
- High-Fidelity Diffusion Face Swapping with ID-Constrained Facial Conditioning (CVPR 2026 · similarity 0.87)
- Zero-Shot Head Swapping in Real-World Scenarios (CVPR 2025 · similarity 0.85)
- Multi-focal Conditioned Latent Diffusion for Person Image Synthesis (CVPR 2025 · similarity 0.83)
- What to Preserve and What to Transfer: Faithful, Identity-Preserving Diffusion-based Hairstyle Transfer (AAAI 2025 · similarity 0.82)
- Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On (CVPR 2024 · similarity 0.82)
- ACFun: Abstract-Concrete Fusion Facial Stylization (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.