Research map / Video / Video generation
MobilePortrait: Real-Time One-Shot Neural Head Avatars on Mobile Devices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MobilePortrait, a lightweight UNet-based one-shot neural head avatar method that uses mixed explicit/implicit keypoints and precomputed pseudo multiview and background features to run over 50 FPS on mobile devices.
Problem
One-shot neural head avatars achieve high quality but at very high computational cost, making real-time on-device deployment on mobile phones impractical.
Approach
Integrate explicit facial keypoints (106 landmarks) with implicit neural keypoints into mixed keypoints, add facial-knowledge losses (landmark and foreground mask prediction), and supply precomputed pseudo multiview features and an inpainted pseudo background so a lightweight UNet can match SOTA quality at under one-tenth the FLOPs.
Key results
- On same-identity reenactment, MobilePortrait (16 GFLOPs) achieves FID 29.2, AKD 1.30, and cross-identity HPD 2.74, leading in AKD and BCI while using under one-tenth the FLOPs of Real3D (610G).
- Runs at 15.8 ms latency on iPhone 14 Pro with 16G FLOPs model (CoreML FP16), and 5.9 ms for the 4G FLOPs variant, enabling over 50 FPS on mobile devices.
- Ablation: removing the proposed facial knowledge losses raises cross-ID HPD from 2.74 to 4.19 and AKD from 1.30 to 1.45.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace mixed keypoints with neural keypoints only (ablation) | MobilePortrait · Same-identity reenactment | FID: 29.2 -> 48.3 | Hurt |
| replace mixed keypoints with facial keypoints only (ablation) | MobilePortrait · Same-identity reenactment | FID: 29.2 -> 33.2 | Hurt |
| remove facial knowledge losses (ablation) | MobilePortrait · Cross-identity reenactment | HPD(C): 2.74 -> 4.19 | Hurt |
| fuse keypoints in the initial transformations instead of using mixed keypoints (ablation) | MobilePortrait · Same-identity reenactment | FID: 29.2 -> 44.7 | Hurt |
| convert keypoints to heatmaps and fuse with a convolutional network (ablation) | MobilePortrait · Same-identity reenactment | FID: 29.2 -> 43.9 | Hurt |
| remove residual optical flow (ablation) | MobilePortrait · Cross-identity reenactment | HPD(C): 2.74 -> 6.2 | Hurt |
| add pseudo-background input to the synthesis network (ablation) | MobilePortrait · Same-identity reenactment | FID: 30.1 -> 29.2 | Helped |
| separately generate foreground and composite it onto the background (ablation) | MobilePortrait · Same-identity reenactment | FID: 29.2 -> 29.7 | Hurt |
| use 0 rather than 4 pseudo multiview inputs (ablation) | MobilePortrait · Same-identity reenactment | FID: 29.2 -> 34.2 | Hurt |
| use 2 rather than 4 pseudo multiview inputs (ablation) | MobilePortrait · Same-identity reenactment | FID: 29.2 -> 31.3 | Hurt |
| use 8 rather than 4 pseudo multiview inputs (ablation) | MobilePortrait · Same-identity reenactment | FID: 29.2 -> 30.2 | Hurt |
| reduce model computation to 4 GFLOPs while retaining MobilePortrait knowledge inputs (ablation) | MobilePortrait · Same-identity reenactment | FID | Helped |
Surprising findings
- Using only neural keypoints or only facial keypoints degrades results more than mixed keypoints, and fusing via transformations or heatmap convolution performs worse than the simple mixed keypoint design.
- Adding more pseudo multiview inputs gives diminishing returns, with 4 views near-optimal in FID and AKD (8 views did not improve FID).
Limitations
- Audio-driven functionality is only a baseline; audio-to-motion module trained on a limited set of the authors' own videos and performs below some methods on Sync-C.
- Cross-identity CSIM is not best among compared methods.
- Real3D was compared using the official CelebVHQ-trained release rather than retrained on the same data.
Experimental setup
Datasets: VFHQ VoxCeleb2 CelebVHQ Talking Head 1K CCv2 HDTF
Benchmarks: Talking Head 1K CCv2 HDTF
Metrics: FID PSNR SSIM AKD HPD AED CSIM BCI Sync-C Sync-D
Baselines: PIRender FaceV2V TPS MCNet Real3D Face2FaceRHO MakeItTalk SadTalker
Builds on: FOMM TPS MCNet MetaPortrait SadTalker
Closest papers by meaning
- RAM-Avatar: Real-time Photo-Realistic Avatar from Monocular Videos with Full-body Control (CVPR 2024 · similarity 0.86)
- EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars (CVPR 2024 · similarity 0.86)
- Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation (CVPR 2025 · similarity 0.85)
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.85)
- High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field (AAAI 2024 · similarity 0.85)
- Real-Time Simulated Avatar from Head-Mounted Sensors (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.