Research map / Image generation / Editing and style transfer
Learning Adaptive Spatial Coherent Correlations for Speech-Preserving Facial Expression Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASCCL, a plug-and-play loss that learns spatial coherent correlations of local visual disparities between source and generated faces, with a disparity-aware weighting, to improve speech-preserving facial expression manipulation.
Problem
Speech-preserving facial expression manipulation (SPFEM) must change emotion while keeping mouth animation tied to spoken content, but paired same-speech different-emotion training data is unavailable for most people.
Approach
Observes that same-content speech with different emotions yields highly correlated local facial motion disparities; learns a contrastive spatial coherent correlation metric over multi-scale ArcFace features and weights regions by disparity magnitude (DAAS) to supervise the generator as an additional loss.
Key results
- On MEAD cross-ID, integrating ASCCL into NED reduces FAD from 4.448 to 4.264, LSE-D from 9.906 to 9.238, and raises CSIM from 0.773 to 0.791.
- On MEAD inter-ID, ICface with ASCCL reduces FAD from 6.795 to 6.672 and LSE-D from 10.083 to 9.431 while raising CSIM from 0.775 to 0.801.
- In a 25-participant user study, NED with ASCCL was preferred in 70% (realism), 69% (emotion similarity), and 73% (mouth shape similarity) of comparisons on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ASCCL supervision | NED · MEAD Cross-ID | FAD: 4.448 -> 4.264 | Helped |
| Add ASCCL supervision | NED · MEAD Cross-ID | LSE-D: 9.906 -> 9.238 | Helped |
| Add ASCCL supervision | NED · MEAD Cross-ID | CSIM: 0.773 -> 0.791 | Helped |
| Add ASCCL supervision | NED · RAVDESS Cross-ID | FAD: 5.412 -> 4.808 | Helped |
| Add ASCCL supervision | NED · RAVDESS Cross-ID | LSE-D: 8.034 -> 7.820 | Helped |
| Add ASCCL supervision | NED · RAVDESS Cross-ID | CSIM: 0.760 -> 0.761 | Helped |
| Pretrain ASCCL with paired data rather than use it untrained (ablation) | NED + ASCCL · MEAD Inter-ID | FAD: 1.509 -> 1.234 | Helped |
| Pretrain ASCCL with paired data rather than use it untrained (ablation) | NED + ASCCL · MEAD Cross-ID | LSE-D: 9.458 -> 9.238 | Helped |
| Apply ASCCL supervision to 3DMM parameters and rendered images rather than 3DMM parameters only (ablation) | NED + ASCCL · MEAD Inter-ID | CSIM: 0.887 -> 0.900 | Helped |
| Apply ASCCL supervision to 3DMM parameters and rendered images rather than rendered images only (ablation) | NED + ASCCL · MEAD Cross-ID | LSE-D: 9.321 -> 9.238 | Helped |
| Add disparity-aware adaptive strategy (DAAS) (ablation) | NED + ASCCL · MEAD Inter-ID | FAD: 1.342 -> 1.234 | Helped |
| Add disparity-aware adaptive strategy (DAAS) (ablation) | NED + ASCCL · MEAD Cross-ID | CSIM: 0.786 -> 0.791 | Helped |
Surprising findings
- Even an untrained ASCCL metric shows high correlation for corresponding regions, and pretraining it on paired data gives further gains.
Limitations
- Relies on paired same-speech different-emotion training data for the ASCCL metric (MEAD), though it is then applied to other persons.
- Gains on some metrics are modest (e.g., cross-ID FAD 4.448 to 4.264) and evaluated only on two SPFEM baselines and two datasets.
Experimental setup
Datasets: MEAD RAVDESS
Benchmarks: MEAD RAVDESS
Metrics: FAD CSIM LSE-D user study ratings
Baselines: NED ICface
Builds on: NED ICface ArcFace InfoNCE-style contrastive loss
Closest papers by meaning
- Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control (ICLR 2025 · similarity 0.88)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.88)
- Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation (AAAI 2024 · similarity 0.87)
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.87)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.87)
- TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.