Research map / Image generation / Editing and style transfer
HiFi-Portrait: Zero-shot Identity-preserved Portrait Generation with High-fidelity Multi-face Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiFi-Portrait, a tuning-free zero-shot identity-preserved portrait generator that fuses multi-reference face features with 3D-aware landmarks via HiFi-Net on SDXL, trained on a newly built 34k-ID dataset.
Problem
Zero-shot identity-preserved portrait generation from multiple reference images of the same ID yields low fidelity and imprecise control of facial expression and pose.
Approach
A face refiner merges local (CLIP) and global (identity extractor) face features into multi-face tokens, and HiFi-Net (ControlNet-derived) fuses these tokens under guidance from DECA-based 3D-aware landmarks that encode the reference ID plus target expression and pose.
Key results
- On WebFace-100, HiFi-Portrait reaches 77.7% Simf versus 72.9% for InstantID and 46.6% for PhotoMaker.
- On InternetFace-300, expression RMSE drops to 16.9 and pose RMSE to 6.2, the best among compared methods.
- Ablation: the face refiner raises Simf from 64.4 (concat) to 65.2, and landmark conditioning in HiFi-Net beats keypoints (65.2 vs 64.5 Simf).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HiFi-Portrait instead of the strongest baseline | SDXL-base-1.0 · WebFace-100 | Simf: 72.9 -> 77.7 | Helped |
| Use HiFi-Portrait instead of the strongest baseline | SDXL-base-1.0 · WebFace-100 | CLIPf: 70.2 -> 72.4 | Helped |
| Use HiFi-Portrait instead of the strongest baseline | SDXL-base-1.0 · WebFace-100 | FID: 114.4 -> 108.5 | Helped |
| Use HiFi-Portrait instead of the strongest baseline | SDXL-base-1.0 · InternetFace-300 | Simf: 69.7 -> 73.1 | Helped |
| Use HiFi-Portrait instead of the strongest baseline | SDXL-base-1.0 · InternetFace-300 | CLIPf: 71.4 -> 71.1 | Hurt |
| Use HiFi-Portrait instead of the strongest baseline | SDXL-base-1.0 · InternetFace-300 | FID: 158.1 -> 163.4 | Hurt |
| Add the face refiner to multi-face features instead of using local features, global features, or their concatenation (ablation) | HiFi-Portrait · WebFace-100 | Simf: 62.3, 63.9, 64.4 -> 65.2 | Helped |
| Add the face refiner to multi-face features instead of using local features, global features, or their concatenation (ablation) | HiFi-Portrait · WebFace-100 | CLIPf: 55.2, 56.8, 57.3 -> 58.9 | Helped |
| Add HiFi-Net with landmarks rather than replacing it with Adapter or conditioning it on keypoints (ablation) | HiFi-Portrait · WebFace-100 | Simf: 61.8, 64.5 -> 65.2 | Helped |
| Add HiFi-Net with landmarks rather than replacing it with Adapter or conditioning it on keypoints (ablation) | HiFi-Portrait · WebFace-100 | Exp: 22.7, 19.8 -> 18.4 | Helped |
| Vary the number of reference images from 1 through 6 at inference (ablation) | HiFi-Portrait · WebFace-100 | Simf: 61.9, 63.8, 64.6, 65.2, 65.0, 64.8 | Mixed |
| Vary the number of reference images from 1 through 6 at inference (ablation) | HiFi-Portrait · WebFace-100 | CLIPf: 52.3, 56.7, 57.1, 58.9, 58.6, 58.5 | Mixed |
Surprising findings
- On InternetFace-300, HiFi-Portrait has a higher FID (163.4) than InstantID (158.1), so it is not best on FID for that set.
Limitations
- Training requires 4 A800 GPUs for 16 days, a large compute budget.
- Dataset is not released, and the construction pipeline depends on many external tools.
Experimental setup
Datasets: VGGFace2-HD IMDb-Face WebFace260M LAION-Face
Benchmarks: WebFace-100 InternetFace-300
Metrics: Simf CLIPf Exp RMSE Pose RMSE FID
Baselines: PhotoMaker ConsistentID FlashFace IP-Adapter InstantID
Builds on: SDXL ControlNet IP-Adapter InstantID DECA
Closest papers by meaning
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.85)
- ID-Sculpt: ID-aware 3D Head Generation from Single In-the-wild Portrait Image (AAAI 2025 · similarity 0.85)
- ID-Patch: Robust ID Association for Group Photo Personalization (CVPR 2025 · similarity 0.85)
- FaithfulFaces: Pose-Faithful Facial Identity Preservation for Text-to-Video Generation (NeurIPS 2026 · similarity 0.85)
- RealisID: Scale-Robust and Fine-Controllable Identity Customization via Local and Global Complementation (AAAI 2025 · similarity 0.84)
- FaceShot: Bring Any Character into Life (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.