Research map / Video / Video segmentation and tracking
Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an enrollment-free, latent-domain defense against puppeteering in AI talking-head videoconferencing: a pose-conditioned contrastive projection (EBL space) plus LSTM temporal fusion detects identity swaps without decoding RGB.
Problem
Puppeteering attacks in neural talking-head videoconferencing let an adversary transmit pose-expression latents that drive a different person's likeness; real-vs-synthetic deepfake detectors fail because all frames are synthetic.
Approach
Re-encode pose-expression latents into a compact biometric-leakage space using a pose-matched contrastive loss (hard negatives share the pose but swap identity), so cosine similarity to the reference identity reflects identity rather than pose; an LSTM over 40-frame similarity windows makes decisions.
Key results
- Average AUC 0.968 across 15 generator/dataset combinations vs 0.944 for Vahdati et al.
- Cross-domain (train NVIDIA VC, test CREMA-D/RAVDESS) average AUC 0.925 vs 0.903 for Vahdati et al.
- Runs at about 75 FPS on an RTX 3090 with under 1M parameters, vs 32 FPS for Vahdati et al.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed method instead of Vahdati et al. | puppeteering detection · Combined datasets, average across generators | AUC: 0.945 to 0.971 | Helped |
| train on NVIDIA VC and evaluate on CREMA-D and RAVDESS | Ours · Cross-domain evaluation | AUC: 0.945 to 0.925 | Mixed |
| remove the biometric leakage MLPs (ablation) | Ours · NVIDIA VC, 3DFaceShop | AUC: 0.827 to 0.966 | Helped |
| replace pose-conditioned contrastive loss with similarity regression (ablation) | Ours · NVIDIA VC, 3DFaceShop | AUC: 0.788 to 0.966 | Helped |
| use a single positive-negative pair per batch (ablation) | Ours · NVIDIA VC, 3DFaceShop | AUC: 0.749 to 0.966 | Helped |
| retain extreme-pose frames (ablation) | Ours · NVIDIA VC, 3DFaceShop | AUC: 0.929 to 0.966 | Helped |
| exclude frames beyond progressively wider yaw thresholds, peaking at ±18° (ablation) | Ours · Puppeteering detection yaw-threshold test | AUC: 0.932; 0.938; 0.953; 0.966; 0.958; 0.942; 0.930; 0.929 | Mixed |
| add eyeglasses (ablation) | Ours · NVIDIA VC, 3DFaceShop | AUC: 0.966 to 0.9393 | Hurt |
| add piercings (ablation) | Ours · NVIDIA VC, 3DFaceShop | AUC: 0.966 to 0.9605 | Hurt |
| add makeup (ablation) | Ours · NVIDIA VC, 3DFaceShop | AUC: 0.966 to 0.9574 | Hurt |
| increase the temporal window from 10 to 40 frames (ablation) | Ours · Temporal window-size evaluation | AUC: ~0.77 to ~0.97 | Helped |
Surprising findings
- Raw cosine similarity in pose-and-expression latent space is pose-dominated: different identities with matched pose can look more similar than the same identity with different pose.
- Adding eyeglasses or piercings lowers AUC modestly, while makeup slightly raises it in the reported ablation.
Limitations
- Fails under extreme head rotation or occlusion.
- Poor lighting or overexposure weakens the biometric signal.
- Motion blur from rapid movement or blinking distorts the latent representation.
Experimental setup
Datasets: NVFAIR NVIDIA VC CREMA-D RAVDESS
Benchmarks: NVFAIR (NVIDIA VC, CREMA-D, RAVDESS)
Metrics: AUC FPS relative error reduction
Baselines: Vahdati et al. Efficient ViT CCE ViT CNN Ensemble TALL SupCon CLRNet LAA-Net
Builds on: Avatar Fingerprinting large-margin cosine loss SupCon
Closest papers by meaning
- Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation (CVPR 2025 · similarity 0.86)
- RealityTest: How People Probe AI Identity and Whether Models Disclose It (NeurIPS 2026 · similarity 0.82)
- From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents (ICML 2026 · similarity 0.82)
- Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes (NeurIPS 2024 · similarity 0.82)
- Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes (NeurIPS 2025 · similarity 0.81)
- FaceShield: Explainable Face Anti-Spoofing with Multimodal Large Language Models (AAAI 2026 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.