Research map / 3D vision / Face and shape
LeGO: Leveraging a Surface Deformation Network for Animatable Stylized Face Generation with One Example
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LeGO, which fine-tunes a FLAME-based surface deformation network with one paired exemplar and CLIP directional losses, plus a Mesh Agnostic Encoder, to produce highly stylized, animatable 3D faces in a desired topology.
Problem
Existing 3D face stylization methods are mostly bounded by 3DMM shape spaces, giving limited stylization, and text or 3D-aware methods fail to output animatable faces in a desired mesh topology.
Approach
Train a surface deformation network on FLAME, then translate its domain to a target style with a single identity/style exemplar pair using hierarchical differentiable rendering, directional CLIP losses and a normal-based style loss; a Mesh Agnostic Encoder (built on NFR encoders) maps arbitrary-topology meshes into its latent space at inference.
Key results
- Ours achieves overall average CLIP score 0.9491 (CLIP-SP 0.9559 / CLIP-IP 0.9273 on simplified meshes), above Deformation Transfer 0.9477, X-mesh 0.8680 and TextDeformer 0.8461.
- In a 34-participant user study, Ours was preferred 60.65% of the time vs 38.17% for Deformation Transfer and under 1% for text-based baselines.
- SIMS sampling gives the lowest average reconstruction error (1.657e-5) versus hybrid (2.144e-5) and vertex-only (3.509e-5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LeGO instead of the strongest baseline, Deformation transfer | stylized face mesh generation · Overall-Average | CLIP average: 0.9477 -> 0.9491 | Helped |
| Add the style loss (ablation) | LeGO · Overall-Average | CLIP average: 0.9470 -> 0.9491 | Helped |
| Use the proposed pseudo-pair style loss instead of direct style-loss comparison (ablation) | LeGO · Overall-Average | CLIP average: 0.9488 -> 0.9491 | Helped |
| Use hierarchical rendering instead of a single canonical view (ablation) | LeGO · Overall-Average | CLIP average: 0.9484 -> 0.9491 | Helped |
| Use SIMS instead of hybrid sampling (ablation) | surface deformation network training · Reconstruction settings average | reconstruction loss (all in e-5): 2.144 -> 1.657 | Helped |
| Use SIMS instead of vertex-only sampling (ablation) | surface deformation network training · Reconstruction settings average | reconstruction loss (all in e-5): 3.509 -> 1.657 | Helped |
| Compare style loss directly using meshes with different expressions (ablation) | LeGO · stylized face animation | animation capability | Hurt |
Surprising findings
- Deformation Transfer scores nearly as high as the proposed method on the CLIP-based averages (0.9477 vs 0.9491), so the metric gap is small.
- Using style loss directly between generated and exemplar normals removes expression and breaks animatability, motivating a pseudo-pair normal loss.
Limitations
- Producing output with the same topology as the input requires a two-stage process with template replacement by the mean face mesh.
- Evaluation relies on a small set of 8 identities and 8 styles, with CLIP-based and user-study metrics only.
Experimental setup
Datasets: FLAME CoMA ICT-FaceKit
Benchmarks: custom stylization eval on FLAME, Loop-subdivided and simplified FLAME meshes
Metrics: CLIP-SP CLIP-IP reconstruction loss (MSE) user preference frequency
Baselines: Deformation Transfer TextDeformer X-mesh
Builds on: FLAME NFR DD3C directional CLIP loss (StyleGAN-NADA-style one-shot adaptation) DiffusionNet
Closest papers by meaning
- 3DToonify: Creating Your High-Fidelity 3D Stylized Avatar Easily from 2D Portrait Images (CVPR 2024 · similarity 0.90)
- Geometry in Style: 3D Stylization via Surface Normal Deformation (CVPR 2025 · similarity 0.89)
- ARMesh: Autoregressive Mesh Generation via Next-Level-of-Detail Prediction (NeurIPS 2025 · similarity 0.88)
- MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers (ICLR 2025 · similarity 0.88)
- En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data (CVPR 2024 · similarity 0.88)
- Styl3R: Instant 3D Stylized Reconstruction for Arbitrary Scenes and Styles (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.