Research map / 3D vision / Face and shape
Electromyography-Informed Facial Expression Reconstruction for Physiological-Based Synthesis and Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EIFER, a CycleGAN-style adversarial framework that reconstructs faces under sEMG electrode occlusion using a 3DMM (FLAME) and learns a bidirectional mapping between expression parameters and muscle activity.
Problem
Facial expression reconstruction and analysis fail when surface EMG electrodes occlude the face, and existing monocular 3D face reconstruction methods are sensitive to such occlusion; the link between facial muscle activity and expressions is not modeled.
Approach
Decouple geometry (SMIRK-style encoders predicting FLAME/BFM parameters) from appearance, and frame appearance reconstruction as unpaired adversarial image-to-image translation with a CycleGAN-like pair of electrode-placing and electrode-removing generators, using occlusion-robust cycle-consistency self-supervision instead of photometric or perceptual losses; then train MLPs for EMG-to-expression
Key results
- EIFER matches MC-CycleGAN on appearance reconstruction (SSIM 0.66, PSNR 19.42, FID 52.56 vs 54.39) and outperforms occluded baselines like SMIRK (FID 275.80).
- For muscle activity prediction from expressions, EIFER achieves the lowest MSE (5.09) and highest Spearman rho (0.38) among compared methods on occluded recordings.
- Qualitatively, EIFER synthesizes expressions from muscle activity that match the recorded frame, while DECA, EMOCAv2 and others struggle under occlusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EIFER's adversarial electrode removal and placement with cycle-consistent geometry constraints instead of MC-CycleGAN | Facial reconstruction · Occlusion-free reference reconstruction | FID: 54.39±24.32 -> 52.56±27.75 | Helped |
| Use EIFER's sEMG-occlusion-aware expression encoder instead of the compared SOTA expression encoders | Expression-to-muscle activity prediction · Occluded recordings | MSE: 6.79±59.74 -> 5.09± 33.97 | Helped |
| Use EIFER's sEMG-occlusion-aware expression encoder instead of DECA | Expression-to-muscle activity prediction · Occluded recordings | RMS: 1.75± 2.79 -> 1.45± 1.73 | Helped |
| Use EIFER's sEMG-occlusion-aware expression encoder instead of DECA | Expression-to-muscle activity prediction · Occluded recordings | SMAPE: 70.65±36.58 -> 67.71±32.54 | Helped |
| Use EIFER's sEMG-occlusion-aware expression encoder instead of SMIRK | Expression-to-muscle activity prediction · Occluded recordings | Spearman ρ: 0.36±0.19 -> 0.38±0.19 | Helped |
| Use EIFER's expression encoder instead of DECA | Expression-to-muscle activity prediction · Restored recordings | SMAPE: 69.17±33.92 -> 67.71±32.54 | Helped |
Surprising findings
- Only SMIRK retains electrode artifacts, while most existing methods implicitly remove electrodes via their appearance models, yet SMIRK still fails the reconstruction when occluded.
- Muscle-activity-to-expression synthesis struggles most with disgusted and surprised expressions due to minimal masseter activity during mouth opening.
Limitations
- Small sample size (N=36) limits generalizability.
- Expression synthesis from muscle activity has difficulty capturing expression intensity and voluntary eye blinks.
- Muscle-activity predictions underestimate amplitude and miss the initial activity surge; evaluation of synthesized expressions is largely visual.
Experimental setup
Datasets: Mimics And Muscles sEMG-facial mimicry dataset (36 participants)
Benchmarks: Mimics And Muscles sEMG dataset
Metrics: SSIM GMSD PSNR MDSI FID MSE RMS SMAPE Spearman rho
Baselines: DECA EMOCAv2 SMIRK Deep3DFace FOCUS MC-CycleGAN
Builds on: SMIRK CycleGAN FLAME MC-CycleGAN
Closest papers by meaning
- 3D Facial Expressions through Analysis-by-Neural-Synthesis (CVPR 2024 · similarity 0.84)
- emg2qwerty: A Large Dataset with Baselines for Touch Typing using Surface Electromyography (NeurIPS 2024 · similarity 0.84)
- emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation (NeurIPS 2024 · similarity 0.83)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.83)
- PTUS: Photo-Realistic Talking Upper-Body Synthesis via 3D-Aware Motion Decomposition Warping (AAAI 2024 · similarity 0.83)
- SpGesture: Source-Free Domain-adaptive sEMG-based Gesture Recognition with Jaccard Attentive Spiking Neural Network (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.