Research map / 3D vision / Human pose estimation
Towards Robust 3D Pose Transfer with Adversarial Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adversarial training with on-the-fly FGM/PGD-generated adversarial pose sources plus a new MAE-style 3D-PoseMAE (multi-scale masking, channel-wise cross-attention) for robust 3D pose transfer, including on raw scans.
Problem
3D pose transfer relies on clean, pre-processed source poses (parametric model fits or skeletons), which are slow to obtain and make pose transfer fragile on noisy point clouds and raw scans.
Approach
Generate adversarial source poses on-the-fly during training using a pose-transfer adversarial loss (inverse distance between generated and ground-truth mesh) with FGM/PGD attacks, combined with a 3D-PoseMAE that uses multi-scale masking and channel-wise cross-attention to focus on extrinsic pose.
Key results
- 3D-PoseMAE reaches PMD 0.6 (x10^-4) on seen and 3.4 on unseen SMPL-NPT poses with clean training, the lowest among compared methods.
- With adversarial training, 3D-PoseMAE gets PMD 16.9 on adversarial samples vs 19.3 for GC-Transformer and 59.0 for NPT.
- On DFAUST raw scans, 3D-PoseMAE achieves PMD 13.70 vs 25.21 for NPT when trained adversarially on SMPL-NPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adversarial training | NPT-MP · SMPL-NPT seen, adversarial samples | PMD (×10−4): 307.1 -> 62.3 | Helped |
| Add adversarial training | NPT · SMPL-NPT seen, adversarial samples | PMD (×10−4): 237.6 -> 59.0 | Helped |
| Add adversarial training | GC-Transformer · SMPL-NPT seen, adversarial samples | PMD (×10−4): 105.2 -> 19.3 | Helped |
| Add adversarial training | 3D-PoseMAE · SMPL-NPT seen, adversarial samples | PMD (×10−4): 77.6 -> 16.9 | Helped |
| Use 3D-PoseMAE instead of NPT | 3D pose transfer · DFAUST raw scan | PMD (×10−4): 25.21 -> 13.70 | Helped |
| Use 3D-PoseMAE instead of NPT | 3D pose transfer · SMPL-NPT Gaussian noise | PMD (×10−4): 12.66 -> 6.13 | Helped |
| Add multi-scale masking to the vanilla architecture (ablation) | 3D-PoseMAE · SMPL-NPT unseen | PMD (×10−4): 4.0 -> 3.8 | Helped |
| Add channel attention after multi-scale masking (ablation) | 3D-PoseMAE · SMPL-NPT unseen | PMD (×10−4): 3.8 -> 3.4 | Helped |
Surprising findings
- Adversarial samples generated against clean-trained NPT raise its PMD from 1.1 to 237.6, showing clean-trained pose transfer models are highly vulnerable.
Limitations
- C&W-based stronger attacks are too slow to integrate into training (over 900x training time).
- Adversarial magnitude must be tuned; too-large perturbations can be removed by statistical outlier removal preprocessing.
Experimental setup
Datasets: SMPL-NPT FAUST DFAUST
Benchmarks: SMPL-NPT DFAUST FAUST
Metrics: PMD
Baselines: DT NPT-MP NPT 3D-CoreNet GC-Transformer
Builds on: GC-Transformer NPT masked autoencoder FGM PGD
Closest papers by meaning
- Neural Pose Representation Learning for Generating and Transferring Non-Rigid Object Poses (NeurIPS 2024 · similarity 0.88)
- 3D Human Pose Estimation in the Wild by Adversarial Learning (CVPR 2018 · similarity 0.87)
- Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects (CVPR 2019 · similarity 0.87)
- A Dual-Augmentor Framework for Domain Generalization in 3D Human Pose Estimation (CVPR 2024 · similarity 0.86)
- Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation (ICLR 2024 · similarity 0.85)
- TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.