Research map / Video / Video generation
X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-NeMo, a zero-shot diffusion portrait animation pipeline that learns a 1D identity-agnostic latent motion descriptor end-to-end and injects it via cross-attention, reducing identity leakage and improving expressiveness.
Problem
Zero-shot portrait animation with diffusion models struggles to transfer extreme or subtle expressions and suffers identity leakage/drift when the reference and driving identities differ.
Approach
Replace spatially aligned ControlNet/PoseGuider motion conditions with a 1D latent motion descriptor learned end-to-end from driving frames and injected through cross-attention, supervised by a dual GAN decoder head, with color/spatial augmentations and reference feature masking.
Key results
- Self-reenactment: X-NeMo achieves L1 0.055, SSIM 0.826, LPIPS 0.168, best among baselines (AniPortrait 0.057/0.812/0.198).
- Cross-reenactment: ID-SIM 0.787 vs. 0.725 for the next best (FYE); EMO-SIM 0.65 vs. 0.52 for X-Portrait.
- Ablation: removing cross-attention control drops ID-SIM to 0.697 and removing augmentation drops it to 0.724.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dual GAN decoder co-training (ablation) | X-NeMo · Cross-Reenactment | EMO-SIM: 0.43 -> 0.65 | Helped |
| train motion encoder end-to-end with the diffusion model (ablation) | X-NeMo · Cross-Reenactment | EMO-SIM: 0.52 -> 0.65 | Helped |
| add reference feature masking (ablation) | X-NeMo · Cross-Reenactment | EMO-SIM: 0.62 -> 0.65 | Helped |
| add color and spatial augmentations (ablation) | X-NeMo · Cross-Reenactment | ID-SIM: 0.724 -> 0.787 | Helped |
| use motion cross-attention instead of spatial ControlNet guidance (ablation) | X-NeMo · Cross-Reenactment | ID-SIM: 0.697 -> 0.787 | Helped |
| use a 512-dimensional motion latent instead of 128 dimensions (ablation) | X-NeMo · Subtle, intricate expressions | motion expressiveness | Helped |
| use a 512-dimensional motion latent instead of 1024 dimensions (ablation) | X-NeMo · Typical driving scenarios | motion expressiveness | No clear effect |
| use 30% reference feature masking rather than 0% (ablation) | X-NeMo · Facial motion transfer | motion detail | Helped |
| avoid excessively high reference feature masking ratios (75% or 95%) (ablation) | X-NeMo · Facial motion transfer | motion detail and identity consistency | Hurt |
Surprising findings
- Removing reference feature masking slightly improves ID-SIM and AED/APD (0.791, 0.039/3.41) versus the full model, though it lowers EMO-SIM.
Limitations
- Trained only on real human videos, so out-of-domain styles such as 3D cartoon characters show artifacts like blurred eyes.
- May struggle with exaggerated expressions absent from the training data.
- Evaluation uses a 256x256 resolution and baselines were finetuned by the authors on their dataset, so comparisons depend on those choices.
Experimental setup
Datasets: HDTF VFHQ NerSemble DFEW MEAD
Benchmarks: DFEW test clips 100 in-the-wild reference portraits
Metrics: L1 SSIM LPIPS ID-SIM AED/APD EMO-SIM
Baselines: PD-FGC LivePortrait X-Portrait Follow-your-Emoji AniPortrait EchoMimic
Builds on: Stable Diffusion AnimateAnyone AniPortrait PD-FGC StyleGAN
Closest papers by meaning
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.89)
- X-Dyna: Expressive Dynamic Human Image Animation (CVPR 2025 · similarity 0.89)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.88)
- EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars (CVPR 2024 · similarity 0.87)
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.86)
- MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.