Research map / 3D vision / Depth and stereo
Deep Non-Rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Supplementary material for a deep NRSfM paper that argues for sequence-level modeling with a context layer and a local (per-sequence) GPA alignment instead of global canonicalization, with extra ablations and visualizations.
Problem
Deep non-rigid structure-from-motion methods typically lift single frames independently and use canonicalization losses that align over whole datasets, which handles sequence structure and motion ambiguity poorly.
Approach
Model the 2D observation as a sequence rather than a set: temporal information is carried only in the weight matrix of a context layer (no additive temporal bias), and alignment is done locally per sequence with a GPA layer rather than globally via a canonicalization loss.
Key results
- On Human3.6M ablation, the original method achieves MPJPE 72.5 versus 95.6 for C3DPO, 109.9 for DNRSfM and 88.3 for PAUL.
- On the short-sequence Kinect dataset, Seq2Seq (0.101 e3D) beats the proposed method (0.166), while on Rug the proposed method reaches 0.193 versus 0.317 for C3DPO.
- On CMU S23/S34 the proposed method (0.168/0.201) is worse than plain Seq2Seq (0.137/0.142), so the local GPA alignment helps on long sequences but not short ones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Ours instead of Seq2Seq | NRSfM reconstruction · Kinect (short sequences) | e3D: 0.101 -> 0.166 | Hurt |
| use Ours instead of the strongest listed baseline, MHR | NRSfM reconstruction · Rug (short sequences) | e3D: 0.306 -> 0.193 | Helped |
| replace Seq2Seq canonicalization loss with GPA Layer (ablation) | Seq2Seq · CMU-S23-All | e3D: 0.137 -> 0.209 | Hurt |
| replace Seq2Seq canonicalization loss with GPA Layer (ablation) | Seq2Seq · CMU-S34-All | e3D: 0.142 -> 0.232 | Hurt |
| replace Seq2Seq canonicalization loss with GPA Layer (ablation) | Seq2Seq · H36M | MPJPE: 79.8 -> 75.6 | Helped |
| use Ours instead of the strongest listed baseline, PAUL | NRSfM reconstruction · H36M | MPJPE: 88.3 -> 72.5 | Helped |
| add an independent projection loss for S' (weight 0.1) (ablation) | Ours · H36M | MPJPE: 72.5 -> 97.1 | Hurt |
| add an independent projection loss for S' (weight 0.4) (ablation) | Ours · H36M | MPJPE: 72.5 -> 119.4 | Hurt |
| add an independent projection loss for S' (weight 1) (ablation) | Ours · H36M | MPJPE: 72.5 -> 112.5 | Hurt |
| add an independent projection loss for S' (weight 1.2) (ablation) | Ours · H36M | MPJPE: 72.5 -> 89.8 | Hurt |
| add an independent projection loss for S' (weight 1.5) (ablation) | Ours · H36M | MPJPE: 72.5 -> 94.8 | Hurt |
| add an independent projection loss for S' (weight 1.8) (ablation) | Ours · H36M | MPJPE: 72.5 -> 94.5 | Hurt |
Surprising findings
- Replacing the canonicalization loss of Seq2Seq with the GPA layer degrades CMU performance (0.137 to 0.209), showing the local alignment is not uniformly better.
- The authors find the sequence reconstruction can be done by the weight-matrix term alone, so the temporal bias term contributes little.
Limitations
- GPA layer underperforms global canonicalization on small datasets or short sequences.
- Local alignment yields no clear gain on small datasets such as CMU Mocap.
- Lacks a stated compute or hardware specification in the supplementary material.
Experimental setup
Datasets: Human3.6M InterHand2.6M CMU Mocap Kinect Rug
Benchmarks: Human3.6M CMU Mocap InterHand2.6M Kinect short sequences Rug
Metrics: MPJPE e3D
Baselines: C3DPO DNRSfM MHR Seq2Seq PAUL
Builds on: C3DPO Seq2Seq
Closest papers by meaning
- Non-Rigid Structure-from-Motion: Temporally-Smooth Procrustean Alignment and Spatially-Variant Deformation Modeling (CVPR 2024 · similarity 0.88)
- VGGSfM: Visual Geometry Grounded Deep Structure From Motion (CVPR 2024 · similarity 0.88)
- MP-SfM: Monocular Surface Priors for Robust Structure-from-Motion (CVPR 2025 · similarity 0.86)
- Structure-from-Motion with a Non-Parametric Camera Model (CVPR 2025 · similarity 0.86)
- Light3R-SfM: Towards Feed-forward Structure-from-Motion (CVPR 2025 · similarity 0.85)
- DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.