Research map / Image generation / Editing and style transfer
ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds TEAD, an LLM-annotated ~50k text-AU-blendshape-expression dataset, trains a CLIP-aligned ExpCLIP encoder, and uses its embeddings with an expression-prompt augmentation scheme to drive text-controlled emotional speech-to-facial-animation generation.
Problem
Speech-driven facial animation usually conveys speech content but lacks flexible, user-friendly emotional style control; existing methods rely on predefined emotion labels or reference images/videos, and no large text-expression paired data exists.
Approach
Use LLMs to annotate emotional text with activated Facial Action Units (converted to blendshape weights) to build a text-expression dataset, align text, expression, and image embeddings in a CLIP-based autoencoder, and train the animation generator with expression prompts extracted from clips while augmenting prompts with blendshape blending and a lip-motion consistency loss.
Key results
- Blendshape augmentation reduced reconstruction MSE on TEAD from 0.051 to 0.012 and on BEAT from 0.069 to 0.021.
- In a 20-volunteer user study, Ours-text scored 3.93 on emotion consistency with text versus 3.37-3.75 for baselines.
- Ours-image scored 3.72 on emotion consistency with video/image, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add blendshape augmentation (ablation) | ExpCLIP · TEAD test set | MSE: 0.051 -> 0.012 | Helped |
| Add blendshape augmentation (ablation) | ExpCLIP · BEAT test set | MSE: 0.069 -> 0.021 | Helped |
| Add text augmentation (ablation) | ExpCLIP · Out-of-domain text prompts | Expression consistency | Helped |
| Add Expression Prompt Augmentation (EPA) (ablation) | Speech-driven facial animation generator · Unseen emotions | Emotional expressiveness | Helped |
| Add style loss (ablation) | EPA-trained speech-driven facial animation generator · Unseen emotions | Emotional impact | Helped |
Limitations
- No quantitative metric for emotion-control precision; evaluation relies on qualitative results and a user study.
- Baseline methods had no released code, so manual annotation of text for templates was needed for comparison.
- Image-prompt extension is described but the framework relies on monocular 3D reconstruction for paired data.
Experimental setup
Datasets: TEAD MEAD-3D BEAT
Benchmarks: TEAD test split MEAD-3D test split BEAT
Metrics: MSE user study score (1-5)
Baselines: EAMM StyleTalk PD-FGC
Builds on: CLIP wav2vec 2.0 FACS
Closest papers by meaning
- MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation (CVPR 2025 · similarity 0.88)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.88)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.87)
- Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation (AAAI 2024 · similarity 0.87)
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.86)
- InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.