Research map / Image generation / Editing and style transfer
UV-IDM: Identity-Conditioned Latent Diffusion Model for Face UV-Texture Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UV-IDM, an identity-conditioned latent diffusion model that generates BFM face UV textures from in-the-wild images via an incomplete-texture condition, and releases the 80K-sample BFM-UV dataset.
Problem
Generating high-fidelity, identity-preserving facial UV textures from single in-the-wild images under occlusions and pose variation, with limited public UV-texture data.
Approach
Treat texture generation as texture completion: extract an incomplete UV texture from the input image (masked by face segmentation and vertex visibility) and inject it into a latent diffusion model via cross-attention through an identity-conditioned module, trained on StyleGAN2-edited triples of in-the-wild, hair-removed, and UV texture images.
Key results
- D-UV-IDM improves over Deep3D on FFHQ with FID 22.65 vs 25.63 and CSIM 0.9428 vs 0.9351.
- H-UV-IDM achieves FID 23.91 on FFHQ vs 27.15 for HRN (0 iterations), though HRN has higher LPIPS and CSIM.
- UV-IDM inference takes about 6 seconds on a P40 GPU versus 150 to 800 seconds for FFHQ-UV, OSTeC, and NextFace.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use incomplete UV texture instead of the original image as the condition (ablation) | D-UV-IDM · FFHQ | LPIPS: 0.1625 -> 0.1575 | Helped |
| use incomplete UV texture instead of the original image as the condition (ablation) | D-UV-IDM · FFHQ | FID: 23.39 -> 22.65 | No clear effect |
| use incomplete UV texture instead of the original image as the condition (ablation) | D-UV-IDM · FFHQ | CSIM: 0.9417 -> 0.9428 | Helped |
| use incomplete UV texture instead of the original image as the condition (ablation) | D-UV-IDM · CelebAMask-HQ | LPIPS: 0.1581 -> 0.1546 | Helped |
| use incomplete UV texture instead of the original image as the condition (ablation) | D-UV-IDM · CelebAMask-HQ | FID: 23.74 -> 24.92 | Hurt |
| use incomplete UV texture instead of the original image as the condition (ablation) | D-UV-IDM · CelebAMask-HQ | CSIM: 0.9512 -> 0.9501 | Hurt |
| use incomplete UV texture instead of the original image as the condition (ablation) | D-UV-IDM · AFLW2000 | LPIPS: 0.1703 -> 0.1651 | Helped |
| use incomplete UV texture instead of the original image as the condition (ablation) | D-UV-IDM · AFLW2000 | FID: 56.80 -> 57.15 | No clear effect |
| use incomplete UV texture instead of the original image as the condition (ablation) | D-UV-IDM · AFLW2000 | CSIM: 0.9233 -> 0.9346 | Helped |
Surprising findings
- Conditioning on the original image instead of the incomplete UV texture gives better FID and CSIM on CelebAMask-HQ in the ablation.
Limitations
- Synthetic training data inherits FFHQ/StyleGAN2 biases and StyleGAN2 editing may lose identity information.
- Quantitative comparison excludes iterative methods (FFHQ-UV, OSTeC, NextFace) due to different 3DMM and overfitting concerns.
- HRN and H-UV-IDM comparison is mixed: HRN has better LPIPS and CSIM.
Experimental setup
Datasets: FFHQ FFHQ-UV CelebAMask-HQ AFLW2000 BFM-UV
Benchmarks: FFHQ CelebAMask-HQ AFLW2000
Metrics: LPIPS FID CSIM inference time
Baselines: Deep3D HRN FFHQ-UV OSTeC NextFace
Builds on: Latent Diffusion Model StyleGAN2 Basel Face Model Deep3D InterFaceGAN
Closest papers by meaning
- FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy (CVPR 2025 · similarity 0.88)
- Multi-focal Conditioned Latent Diffusion for Person Image Synthesis (CVPR 2025 · similarity 0.87)
- PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing (CVPR 2025 · similarity 0.86)
- Text-Guided 3D Face Synthesis - From Generation to Editing (CVPR 2024 · similarity 0.86)
- Single Mesh Diffusion Models with Field Latents for Texture Generation (CVPR 2024 · similarity 0.86)
- Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.