Research map / 3D vision / Face and shape
OFER: Occluded Face Expression Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OFER, two conditional DDPMs that generate FLAME shape and expression coefficients from an occluded face image, plus a learned ranking network that selects the best identity sample, evaluated on NoW, Dey et al. and a new CO-545 occlusion benchmark.
Problem
Single-image 3D face reconstruction under occlusion is ill-posed and multi-hypothesis, yet most methods output a single deterministic solution and existing diverse methods produce unrealistic faces.
Approach
Use two diffusion models to sample a distribution of FLAME shape and expression coefficients conditioned on the image, and a softmax-based ranking network that scores sampled identity shapes to pick one consistent identity across expression variants.
Key results
- On NoW validation, OFER-rank achieves 1.21 mean MSE over both subsets versus 1.31 for FOCUS and 1.78 for Diverse3D.
- On CO-545, OFER reduces expression sample error (occluded) from 4.72 (Diverse3D) to 2.78 RMSE and CSE from 0.30 to 0.17.
- Ablation: the ranking network lowers ranked-sample error versus average of samples (e.g., OFER(100) ranked mean 1.21 vs average 1.25 on NoW; FLAME mixes ranked mean 1.31 vs average 1.79).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OFER rather than Diverse3D for expression reconstruction | OFER · Dey Dataset, expression occluded images | RMSE: 7.46 -> 3.48 | Helped |
| Use OFER rather than Diverse3D for expression reconstruction | OFER · CO-545, expression occluded images | RMSE: 4.72 -> 2.78 | Helped |
| Use OFER rather than Diverse3D for neutral reconstruction | OFER · CO-545, neutral occluded images | RMSE: 2.16 -> 1.95 | Helped |
| Use OFER rather than Diverse3D for closest-sample error | OFER · CO-545 | CSE RMSE: 0.30 -> 0.17 | Helped |
| Use OFER rather than Diverse3D for expression diversity | OFER · CO-545 | ASD-O: 0.16 -> 0.18 | Helped |
| Use OFER rather than Diverse3D for in-distribution plausibility | OFER · Mask occlusions | ODE: 0.95 -> 0.002 | Helped |
| Add IdRank selection to generated shape samples (ablation) | OFER(100) · NoW validation | median MSE: 1.02 -> 0.98 | Helped |
| Use OFER-generated samples rather than random FLAME samples (ablation) | OFER shape search · NoW validation | ideal lowest-error median MSE: (1.00 - 0.90) -> 0.81 | Helped |
| Add OFER samples to a candidate pool that is 50% random FLAME samples (ablation) | Mixed candidate ranking · NoW validation | ranked-sample median MSE: 1.33 -> 1.08 | Helped |
| Add OFER samples to a candidate pool that is 20% OFER samples (ablation) | Mixed candidate ranking · NoW validation | ranked-sample median MSE: 1.60 -> 1.34 | Helped |
Surprising findings
- Ranking gains are modest on the full NoW set: ranked OFER-rank (1.21 mean) is close to the average of all samples (1.25), while the oracle best (OFER-min) reaches 1.05.
Limitations
- Trained on only a small subset of available 3D supervision, so expressions may lack detail.
- Ranking selects a good but not necessarily the absolute best sample; softmax scoring dilutes scores among high-quality samples.
- Evaluation uses synthetic occlusions on CO-545 and the test split lacks occluded/unoccluded labels.
Experimental setup
Datasets: Stirling FaceWarehouse LYHM Florence FaMoS CoMA NoW
Benchmarks: NoW validation (occluded/unoccluded) CO-545 Dey et al. dataset Erakiotan et al. hard-occlusion dataset
Metrics: median MSE mean MSE std MSE sample error closest sample error ASD-V ASD-O STD-S ODE RMSE
Baselines: MICA Diverse3D FOCUS EMOCA DECA Deep3D TokenFace FLAME random sampling
Builds on: DDPM MICA FLAME Diverse3D
Closest papers by meaning
- Generating Diverse 3D Reconstructions From a Single Occluded Face Image (CVPR 2022 · similarity 0.91)
- Learning Complete 3D Morphable Face Models From Images and Videos (CVPR 2021 · similarity 0.89)
- 3D Facial Expressions through Analysis-by-Neural-Synthesis (CVPR 2024 · similarity 0.88)
- Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction (ICLR 2026 · similarity 0.88)
- O^2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model (AAAI 2024 · similarity 0.87)
- TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.