AI Paper Atlas

Research map / 3D vision / Face and shape

MeshTalk: 3D Face Animation From Speech Using Cross-Modality Disentanglement

Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando de la Torre, Yaser Sheikh

ICCV 2021 · 3D vision › Face and shape · approach: Speech-driven 3D facial animation · 259 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add audio to the encoder input (with l2 loss) (ablation)MeshTalk latent learning · Latent space strategies (in-house test set)autoregressive model perplexity: 1.853 -> 1.879No clear effect
add audio to the encoder input (with l2 loss) (ablation)MeshTalk latent learning · Latent space strategies (in-house test set)reconstruction error (mm): 1.156 -> 1.124No clear effect
add cross-modality loss on top of expression+audio encoder (ablation)MeshTalk latent learning · Latent space strategies (in-house test set)autoregressive model perplexity: 1.879 -> 1.669Improved
add cross-modality loss on top of expression+audio encoder (ablation)MeshTalk latent learning · Latent space strategies (in-house test set)reconstruction error (mm): 1.124 -> 1.244Hurt
use audio and expression encoder with cross-modality loss instead of expression-only encoder (ablation)MeshTalk latent learning · Latent space strategies (in-house test set)autoregressive model perplexity: 1.853 -> 1.669Improved
use categorical instead of continuous latent space (ablation)MeshTalk · Latent space type (in-house test set)vertex error (mm): 1.975 -> 1.244Improved
use categorical instead of continuous latent space (ablation)MeshTalk · Latent space type (in-house test set)lip error (mm): 4.578 -> 3.184Improved
use MeshTalk instead of VOCAaudio-driven face animation · Lip sync vs VOCA (in-house test set)lip vertex error (mm): 3.720 -> 3.184Improved
replace DeepSpeech features in VOCA with Mel spectrograms and our audio encoder (ablation)VOCA · Lip sync vs VOCA (in-house test set)lip vertex error (mm): 3.720 -> 3.472Improved
use MeshTalk instead of VOCAaudio-driven face animation · Perceptual study, full-face% ours better or equal: 75.3%Improved
use MeshTalk instead of VOCAaudio-driven face animation · Perceptual study, upper face% ours better or equal: 66.4%Improved
compare MeshTalk with tracked ground truthMeshTalk · Perceptual study, upper face% ours better or equal: 31.5%Hurt

Later papers that compared with this method

2 later accepted papers report MeshTalk as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy