Research map / 3D vision / Face and shape

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

NeurIPS 2026 · 3D vision › Face and shape · approach: Speech-driven 3D facial animation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an articulation-aware speech-driven 3D facial animation framework that predicts three directional lip-jaw articulatory motions (spreading, opening, protrusion) via a key-value Speech-Articulatory Memory and composes them over mesh topology with a GCN+TCN module (TAC).

Problem

Speech-driven 3D facial animation often yields speech-inconsistent visible lip/jaw articulation because the acoustic-to-motion mapping is one-to-many and existing methods regress holistic vertex motion without modeling structured articulatory constraints.

Approach

Decompose visible articulation into three directional motions (spreading/rounding, opening/closing, protrusion/retraction) predicted by per-direction value memories retrieved via acoustic keys (SAM), then compose them into full facial motion using topology-aware graph convolution and temporal convolution with a Laplacian consistency loss (TAC).

Key results

• On VOCASET, the method achieves FVE 0.771, LVE 0.235, FDD 0.097, LDTW 0.132, the lowest FVE/LVE/LDTW among compared methods.
• On TFHP, FVE 0.100 and LVE 0.108 are lowest, while FDD (0.578) is second to DiffPoseTalk (0.321).
• User study with 24 valid participants shows average preference of 70.3% for lip sync and 70.7% for realism against seven competitors.

Surprising findings

• On TFHP, the method's FDD (upper-face dynamics) is not best; DiffPoseTalk achieves 0.321, indicating the articulation focus does not help upper-face dynamics.

Limitations

• Does not explicitly model upper-face dynamics, leading to weaker FDD on TFHP.
• Relies on directional decomposition defined from a small set of lip anchors, so articulation beyond these anchors is not explicitly modeled.
• Evaluation on only a few datasets with FLAME-based topology and head pose removed.

Experimental setup

Datasets: VOCASET TFHP BIWI HDTF

Benchmarks: VOCASET TFHP BIWI

Metrics: FVE LVE FDD LDTW visible articulatory distance error visible articulatory velocity error pairwise user preference

Baselines: FaceFormer CodeTalker SelfTalk Imitator ScanTalk UniTalker MemoryTalker StreamingTalker Mimic DiffPoseTalk ARTalk

Builds on: FaceFormer HuBERT Wav2Lip Savitzky-Golay filter

Closest papers by meaning

• Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.90)
• ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.90)
• Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.90)
• Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation (AAAI 2024 · similarity 0.88)
• DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.88)
• Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation (CVPR 2025 · similarity 0.88)