Research map / 3D vision / Face and shape
Neural Pose Representation Learning for Generating and Transferring Non-Rigid Object Poses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns an identity-agnostic keypoint-based hybrid pose representation for 3D deformable meshes, using an implicit Jacobian-field deformation decoder, per-identity self-supervised refinement, and a cascaded diffusion model to generate new poses for transfer to other identities.
Problem
Transferring and generating non-rigid 3D object poses while disentangling pose from identity, without skeletons, correspondences, or target-identity pose variations.
Approach
Represents pose as a set of 100 FPS-sampled keypoints with learned features, decodes it through an implicit deformation field that predicts per-face Jacobians solved by a differentiable Poisson layer, adds a self-supervised per-identity refinement adapter, and trains cascaded diffusion models over the keypoint representation for pose generation.
Key results
- On ormthings, FID 1.11 (x10^-2) and ResNet accuracy 78.72% versus NJF 11.33 FID and 64.43% and ZPT 19.88 FID.
- On SMPL, PMD 0.13 (x10^-3), lower than SPT's 0.28 and NJF's 2.55.
- Ablation: Jacobian representation improves generated-pose FID from 9.52 to 4.40 (x10^-2) and refinement further to 4.22.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed Jacobian-based pose representation and per-identity refinement rather than NJF | pose transfer · ormthings | FID (×10^-2): 11.33 -> 1.11 | Helped |
| Use the proposed Jacobian-based pose representation and per-identity refinement rather than NJF | pose transfer · ormthings | ResNet accuracy (%): 64.43 -> 78.72 | Helped |
| Use the proposed method rather than SPT | pose transfer · SMPL | PMD (×10^-3): 0.28 -> 0.13 | Helped |
| Use the proposed method rather than SPT | pose transfer · SMPL | ResNet accuracy (%): 75.38 -> 79.09 | Helped |
| Represent shapes with Jacobian fields instead of vertex coordinates (ablation) | pose transfer · ormthings | FID (×10^-2): 3.52 -> 1.17 | Helped |
| Add per-identity refinement to the Jacobian-based method (ablation) | pose transfer · ormthings | ResNet accuracy (%): 75.13 -> 78.72 | Helped |
| Represent shapes with Jacobian fields instead of vertex coordinates (ablation) | generated poses · ormthings | FID (×10^-2): 9.52 -> 4.40 | Helped |
| Add per-identity refinement to the Jacobian-based method (ablation) | generated poses · ormthings | ResNet accuracy (%): 75.82 -> 78.81 | Helped |
| Reduce the number of keypoints from 100 to 50 (ablation) | pose transfer · ormthings | FID (×10^-2): 0.72 -> 0.83 | No clear effect |
| Reduce the number of keypoints from 100 to 10 (ablation) | pose transfer · SMPL | PMD (×10^-3): 0.13 -> 0.20 | No clear effect |
Limitations
- Requires preprocessing for meshes with multiple disconnected components or triangulation defects.
- Assumes a template mesh of the target shape is known for pose transfer.
- Models are trained separately per identity.
Experimental setup
Datasets: ormthings (DeformingThings4D animals) SMPL/VPoser synthetic humans Mixamo characters
Benchmarks: ormthings SMPL human dataset Mixamo characters
Metrics: FID KID ResNet classification accuracy PMD
Baselines: NJF SPT ZPT vertex-based variant of ours
Builds on: Point Transformer Poisson surface editing / Jacobian fields cascaded diffusion models Morphing Humans (NJF)
Closest papers by meaning
- Towards Robust 3D Pose Transfer with Adversarial Learning (CVPR 2024 · similarity 0.88)
- SPAMs: Structured Implicit Parametric Models (CVPR 2022 · similarity 0.88)
- Learning Implicit Representation for Reconstructing Articulated Objects (ICLR 2024 · similarity 0.87)
- REACTO: Reconstructing Articulated Objects from a Single Video (CVPR 2024 · similarity 0.86)
- NOPE: Novel Object Pose Estimation from a Single Image (CVPR 2024 · similarity 0.86)
- Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.