Research map / 3D vision / Face and shape
TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TEASER, a hybrid 3DMM-plus-appearance-token model that uses a multi-scale tokenizer and token-guided neural renderer to improve single-image facial expression reconstruction.
Problem
Single-image 3D face reconstruction fails to capture fine-grained expressions (irregular mouth shapes, exaggerated or asymmetric movements), partly because photometric supervision from differentiable renderers is weak.
Approach
Predict FLAME explicit parameters plus an implicit multi-scale appearance token; a token-guided U-Net renderer (AdaIN plus zero-conv token decoder) turns the mesh rendering into a realistic face image to give stronger photometric supervision, with a token cycle loss, pose-dependent landmark loss, and mouth/eye region loss.
Key results
- On LRS3, TEASER reaches FID 19.41, CSIM 0.804, PSNR 30.67 and AED 0.114, improving over SMIRK (FID 25.39, PSNR 29.14).
- On HDTF, TEASER's AED is 0.036 versus 0.138 for SMIRK, and its flicker is 0.662 versus 1.484.
- On FaceWarehouse, TEASER's mean error is 3.87 versus 3.98 for SMIRK, a small margin; the paper states it is comparable on shape benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SMIRK with TEASER | FLAME-based facial reconstruction · LRS3 | LPIPS: 0.109 -> 0.077 | Helped |
| Replace SMIRK with TEASER | FLAME-based facial reconstruction · HDTF | FID: 35.27 -> 30.19 | Helped |
| Replace SMIRK with TEASER | FLAME-based facial reconstruction · LRS3 | warp-error: 1.245 -> 0.801 | Helped |
| Add a single-scale appearance token T' (ablation) | SMIRK baseline · LRS3 | LPIPS: 0.109 -> 0.095 | Helped |
| Use a multi-scale appearance token T instead of T' (ablation) | SMIRK baseline with appearance token · LRS3 | LPIPS: 0.095 -> 0.089 | Helped |
| Add token consistency loss (ablation) | Multi-scale token model · LRS3 | LPIPS: 0.089 -> 0.087 | Helped |
| Add region loss (ablation) | Token-consistency model · LRS3 | LPIPS: 0.087 -> 0.086 | Helped |
| Add zero-convolution layers to the token decoder (ablation) | Region-loss model · LRS3 | LPIPS: 0.086 -> 0.078 | Helped |
| Add pose-dependent landmark loss (ablation) | Zero-convolution model · LRS3 | LPIPS: 0.078 -> 0.077 | Helped |
Surprising findings
- Simply enlarging SMIRK's renderer (to 56.4M parameters) did not improve its metrics, suggesting the gain comes from the token design rather than renderer capacity.
Limitations
- Misalignment on extreme poses and heavily occluded faces inherited from FLAME-based reconstruction.
- Cannot generate complex occlusions such as hands, microphones, bangs or accessories.
- The hard pose-dependent landmark mask ignores eyebrow and other landmarks; a smoother confidence score is left to future work.
Experimental setup
Datasets: FFHQ CelebA LRS3
Benchmarks: LRS3 HDTF NoW Stirling FaceWarehouse
Metrics: LPIPS FID CSIM PSNR AED APD warp-error flicker median/mean/std scan-to-mesh error
Baselines: SMIRK 3DDFA-v3 DECA EMOCA Deep3DFace 3DDFA-V2
Builds on: SMIRK FLAME ControlNet AdaIN MICA
Closest papers by meaning
- 3D Facial Expressions through Analysis-by-Neural-Synthesis (CVPR 2024 · similarity 0.90)
- OFER: Occluded Face Expression Reconstruction (CVPR 2025 · similarity 0.86)
- Learning Adaptive Spatial Coherent Correlations for Speech-Preserving Facial Expression Manipulation (CVPR 2024 · similarity 0.86)
- High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field (AAAI 2024 · similarity 0.85)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.85)
- AUEditNet: Dual-Branch Facial Action Unit Intensity Manipulation with Implicit Disentanglement (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.