Research map / Label-efficient and robust learning / Self-supervised and contrastive

Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation

CVPR 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervised speech and brain decoding · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Wav2Sem, a plug-and-play module that extracts sentence-level semantic features from audio (TCN + Transformer aligned to BERT) and fuses them into self-supervised audio features to decouple near-homophonic syllables for 3D speech-driven facial animation.

Problem

Self-supervised audio encoders (HuBERT, Wav2Vec 2.0) couple near-homophonic syllables with different lip shapes in feature space, causing averaging effects and lip-sync errors in 3D speech-driven facial animation.

Approach

Pretrain an audio-only semantic encoder (TCN + Transformer) to regress BERT sentence embeddings on LibriSpeech-960, then fuse its pooled output into frame-level self-supervised audio features via linear layers and summation, so the downstream animation model gets globally contextualized audio features without text input.

Key results

• Integrating Wav2Sem into six speech-driven facial animation models reduces LVE on VOCASET, e.g., FaceFormer LVE from 4.1090 to 3.9891 (×10^-5) and CodeTalker from 3.9445 to 3.8714 (×10^-5) with Wav2Sem variants.
• On TIMIT phoneme recognition, Wav2Vec 2.0 test PER improves from 8.3 to 8.0 and HuBERT from 8.0 to 7.8 with Wav2Sem-c.
• Word-level L2 feature distance for homophone pairs rises from 0.0397 to 0.0701 with Wav2Vec 2.0 and from 0.2689 to 0.2909 with HuBERT.

Measured effects reported in the paper

Surprising findings

• Wav2Sem-derived audio semantic features outperform directly injecting BERT text-derived sentence embeddings in most VOCASET ablation settings, attributed to domain gap between VOCASET and LibriSpeech/BooksCorpus.
• Improvements in MVE and LVE are often small (typically around 1-5%), suggesting modest gains relative to baselines.

Limitations

• Gains over baselines are modest in several metrics, and some variants (e.g., Wav2Sem-m on FaceFormer MVE/FDD on VOCASET) are worse than baseline.
• Decoupling evaluated on a small set of 90 word pairs and qualitative examples; the homophone set was generated with GPT-4o mini.
• Ablation choice between CLS and mean pooling depends on dataset size, with no principled selection rule.

Experimental setup

Datasets: LibriSpeech-960 BIWI VOCASET TIMIT CMU Pronouncing Dictionary

Benchmarks: VOCASET BIWI TIMIT

Metrics: LVE MVE FDD PER L2 feature distance user study score (5-point)

Baselines: VOCA FaceFormer CodeTalker UniTalker FaceDiffuser LG-LDM BERT-based semantic features

Builds on: Wav2Vec 2.0 HuBERT BERT FaceFormer CodeTalker

Closest papers by meaning

• Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation (AAAI 2024 · similarity 0.88)
• Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.88)
• Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation (NeurIPS 2026 · similarity 0.88)
• ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.87)
• KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.87)
• FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio (CVPR 2024 · similarity 0.87)