Research map / Video / Video generation
FaceFormer: Speech-Driven 3D Facial Animation With Transformers
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FaceFormer instead of VOCA or MeshTalk | speech-driven 3D facial animation · BIWI-Test-A | Lip Vertex Error: VOCA: 7.6427; MeshTalk: 6.7436; FaceFormer: 5.3742 | Helped |
| use FaceFormer instead of VOCA or MeshTalk | speech-driven 3D facial animation · BIWI-Test-B | perceived realism and lip synchronization: 83.85\pm3.76; 82.64\pm3.77; 83.33\pm4.07; 80.56\pm5.22 | Helped |
| use FaceFormer instead of VOCA or MeshTalk | speech-driven 3D facial animation · VOCA-Test | perceived realism and lip synchronization: 77.92\pm7.94; 77.08\pm7.32; 82.92\pm2.60; 82.08\pm3.15 | Helped |
| add encoder transformer self-attention to the pretrained TCN (ablation) | FaceFormer · BIWI | lip synchronization quality | Helped |
| initialize the encoder with wav2vec 2.0 weights (ablation) | FaceFormer · BIWI | face movement quality | Helped |
| use an autoregressive Transformer decoder instead of a fully connected layer (ablation) | FaceFormer · BIWI | lip motion stability and synchronization | Helped |
| use a Transformer decoder instead of an LSTM decoder (ablation) | FaceFormer · BIWI | temporal coherence of lip motions | Helped |
| add alignment bias to cross-modal attention (ablation) | FaceFormer · BIWI | facial expression quality | Helped |
| use temporal bias plus periodic positional encoding instead of original sinusoidal positional encoding (ablation) | FaceFormer · Long TED audio clips | perceived realism: 57.78\% \pm 16.78\% | Helped |
| use temporal bias plus periodic positional encoding instead of original sinusoidal positional encoding (ablation) | FaceFormer · Long TED audio clips | perceived lip synchronization: 62.22\% \pm 10.18\% | Helped |
| replace temporal bias plus periodic positional encoding with original sinusoidal positional encoding (ablation) | FaceFormer · Long audio sequences | lip motion stability | Hurt |
| replace temporal bias plus periodic positional encoding with ALiBi (ablation) | FaceFormer · VOCASET | facial expression continuity | Hurt |
Closest papers by meaning
- Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.91)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.90)
- Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation (AAAI 2024 · similarity 0.89)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.89)
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.88)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.