Research map / 3D vision / Human pose estimation

FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance Head-pose and Facial Expression Features

CVPR 2024 · 3D vision › Human pose estimation · approach: Temporal video pose estimation · 24 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FSRT, a transformer encoder-decoder that builds a set-latent source face representation and renders query pixels conditioned on keypoints and a learned expression vector, enabling cross-reenactment with factorized appearance, pose, and expression.

Problem

Face reenactment: transferring head pose and facial expressions from a driving video to a source face image (including cross-identity), where CNN optical-flow warp-and-refine methods suffer from shape leakage and temporal inconsistency.

Approach

Adapt Scene Representation Transformers to faces: encode unposed source images into a set-latent representation learned self-supervised, decode each pixel with a transformer conditioned on driving keypoints and an expression vector, with color-jitter/crop invariance and VICReg-style statistical regularization to separate appearance, pose, and expression.

Key results

• Multi-source model (Ours/2-Src) reaches SSIM 0.7891, PSNR 25.00 on VoxCeleb self-reenactment, the best among compared methods.
• Single-source model reaches SSIM 0.7576 and PSNR 23.67, competitive with TSMM (0.7660 / 23.76) and ahead of FOMM and DaGAN on L1.
• In a 20-scene cross-reenactment user study, users preferred FSRT over DaGAN in relative motion 98% of the time and over FOMM 97%.

Measured effects reported in the paper

Surprising findings

• The small-decoder variant achieves self-reenactment performance close to the reference model while raising throughput from 11 to 23 fps, suggesting the set-latent representation is efficient to decode.

Limitations

• Struggles to generate out-of-distribution expressions such as sticking out the tongue or looking back.
• Background and hair details are sometimes slightly reduced compared to CNN-based methods.
• Trained only in self-reenactment regime since ground truth is needed; cross-reenactment generalization relies on regularization.

Experimental setup

Datasets: VoxCeleb

Benchmarks: VoxCeleb test set

Metrics: SSIM PSNR L1 AKD user preference

Baselines: FOMM DaGAN DaGAN/dv2 OSFS TSMM DPE

Builds on: Scene Representation Transformer FOMM VICReg DaGAN

Closest papers by meaning

• Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.86)
• X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention (ICLR 2025 · similarity 0.85)
• Learning Dense Correspondence for NeRF-Based Face Reenactment (AAAI 2024 · similarity 0.85)
• EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars (CVPR 2024 · similarity 0.85)
• RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.85)
• Enhancing Identity-Deformation Disentanglement in StyleGAN for One-Shot Face Video Re-Enactment (AAAI 2025 · similarity 0.85)