Research map / Video / Video generation
FaceShot: Bring Any Character into Life
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FaceShot, a training-free portrait animation framework that uses diffusion-feature landmark matching with appearance guidance and coordinate-based landmark retargeting to drive any character from any video via a landmark-driven model (MOFA-Video).
Problem
Portrait animation relies on facial landmark detectors that fail to generalize to non-human characters (emojis, animals, toys), and existing methods struggle to capture subtle facial motions.
Approach
Use semantic correspondences of Stable Diffusion features, injected with appearance priors via IP-Adapter and an appearance gallery, to match landmarks on any character; then retarget driving-video motion to those landmarks using part-wise rectangular coordinate systems, and feed the resulting landmark sequence into a pre-trained landmark-driven animator.
Key results
- On CharacBench, FaceShot reaches ArcFace 0.848 versus 0.695 for its base MOFA-Video, and Point-Tracking 6.935 versus 14.985.
- Landmark matching achieves NME 8.569 on CharacBench, beating DIFT (11.448), Uni-Pose (13.731) and STAR (24.530).
- FaceShot adds only about 119 ms overhead to MOFA-Video for 50 frames (79.54 s total vs 79.42 s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MOFA-Video's landmark input with FaceShot's reposed landmarks | MOFA-Video · CharacBench | ArcFace: 0.695 -> 0.848 | Helped |
| replace MOFA-Video's landmark input with FaceShot's reposed landmarks | MOFA-Video · CharacBench | HyperIQA: 52.272 -> 53.723 | Helped |
| replace MOFA-Video's landmark input with FaceShot's reposed landmarks | MOFA-Video · CharacBench | Point-Tracking: 14.985 -> 6.935 | Helped |
| use k=5 target images rather than k=1 (ablation) | FaceShot landmark matching · 300W | NME: 12.801 -> 7.009 | Helped |
| use k=10 target images rather than k=5 (ablation) | FaceShot landmark matching · 300W | NME: 7.009 -> 6.343 | Helped |
| use k=15 target images rather than k=10 (ablation) | FaceShot landmark matching · 300W | NME: 6.343 -> 6.267 | Helped |
| use k=50 target images rather than k=10 (ablation) | FaceShot landmark matching · 300W | NME: 6.343 -> 6.252 | Helped |
| use k=1000 target images rather than k=10 (ablation) | FaceShot landmark matching · 300W | NME: 6.343 -> 6.104 | Helped |
| use FaceShot appearance-guided landmark matching instead of DIFT | FaceShot landmark matching · CharacBench | NME: 11.448 -> 8.569 | Helped |
| use FaceShot coordinate-based landmark retargeting instead of Deep3D | FaceShot landmark retargeting · CharacBench | Point-Tracking: 8.282 -> 6.935 | Helped |
Surprising findings
- Several baselines (AniPortrait, MegActor, LivePortrait) fail to produce animations for a substantial fraction of characters (16.67% to 39.13% failure rates).
Limitations
- Non-human target landmarks are annotated manually, so the benchmark relies on human labeling.
- Retargeting is a hand-designed heuristic with no learned parameters, which the authors note limits control over expressions.
- Evaluation uses a single 46-character benchmark with 24 driving videos and a small user study of 20 volunteers on 15 cases.
Experimental setup
Datasets: CharacBench 300W RAVDESS HDTF
Benchmarks: CharacBench 300W HDTF
Metrics: ArcFace score HyperIQA LAION Aesthetic Point-Tracking NME user preference (Motion, Identity, Overall)
Baselines: MOFA-Video X-Portrait FaceVid2Vid FADM Follow Your Emoji LivePortrait MegActor AniPortrait DIFT STAR Uni-Pose Deep3D
Builds on: MOFA-Video Stable Diffusion IP-Adapter DIFT Deep3D
Closest papers by meaning
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.88)
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.88)
- Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · similarity 0.87)
- ReMask-Animate: Refined Character Image Animation Using Mask-Guided Adapters (AAAI 2025 · similarity 0.87)
- EchoShot: Multi-Shot Portrait Video Generation (NeurIPS 2025 · similarity 0.87)
- MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.