Research map / 3D vision / Face and shape
S3O: A Dual-Phase Approach for Reconstructing Dynamic Shape and Skeleton of Articulated Objects from Single Monocular Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S3O, a two-phase (coarse shape then motion and fine shape with EM-style skeleton learning) method that reconstructs shape and physical skeleton of articulated animals from a single monocular video without templates or camera poses, and introduces the PlanetZoo dataset.
Problem
Reconstructing dynamic articulated objects (shape, motion, camera, and skeleton) from a single monocular video with limited viewpoints, without predefined parametric models, camera poses, or keypoint annotations, and at lower computational cost.
Approach
A two-phase optimization: first learn a coarse shape from a canonical frame's 2D skeleton lifted to 3D, then alternate EM-style between mesh/motion updates (E-step) and skeleton adaptation via bone shifting, skeleton growing, and physically constrained refinement (M-step), with a Dynamic Rigidity loss replacing ARAP.
Key results
- On DAVIS and PlanetZoo 2D keypoint transfer, S3O averages 80.4 PCK-T on DAVIS vs 70.8 for LASR, and 66.2 on PlanetZoo vs 62.8 for LASR, with ~3.8-4.2 GPU hours vs ~9-12 for baselines.
- On AMA-Human (samba, swing), S3O reports average Chamfer distance 12.9 vs 14.6 for BANMo and F@2% 55.8 vs 53.9, with 4.9 vs 12.6 GPU hours.
- The paper reports S3O reduces training time by roughly 60% versus state of the art.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use S3O instead of LASR | 2D keypoint transfer · DAVIS (Ave.) | 2D keypoint transfer accuracy: 70.8 -> 80.4 | Helped |
| Use S3O instead of ViSER | 2D keypoint transfer · DAVIS (Ave.) | training time (GPU hours on A100): 10.4 -> 3.8 | Helped |
| Use S3O instead of LASR | 2D keypoint transfer · PlanetZoo (Ave.) | 2D keypoint transfer accuracy: 62.8 -> 66.2 | Helped |
| Use S3O instead of ViSER | 2D keypoint transfer · PlanetZoo (Ave.) | training time (GPU hours on A100): 12.1 -> 4.2 | Helped |
| Use S3O instead of BANMo | 3D reconstruction · AMA-samba | Chamfer distance: 15.3 -> 12.9 | Helped |
| Use S3O instead of BANMo | 3D reconstruction · AMA-swing | F@2%: 54.8 -> 56.4 | Helped |
| Use S3O instead of BANMo | 3D reconstruction · AMA-Human (Ave.) | training time (GPU hours on A100): 12.6 -> 4.9 | Helped |
| Remove DINO feature loss during the motion and fine shape phase (ablation) | S3O · motion and fine shape phase | shape detail | Helped |
| Use Dynamic Rigidity instead of ARAP or no regularization (ablation) | S3O reconstruction · shape physical plausibility | physical plausibility | Helped |
Surprising findings
- Using estimated bone SE(3) transforms for bone motion performed worse than 2D optical flow warping, which the authors attribute to unstable bone pose predictions after skeleton updates.
Limitations
- Training remains computationally expensive, which the authors say makes methods impractical for everyday use.
- Bone motion via SE(3) estimates was less reliable than optical flow, and the method depends on mask and optical flow quality.
- Relies on DINO features and part clustering in the coarse phase, and the DINO feature loss is removed in the fine phase because of limited accuracy.
Experimental setup
Datasets: DAVIS BADJA PlanetZoo AMA-Human
Benchmarks: DAVIS BADJA PlanetZoo AMA-Human
Metrics: PCK-T 2D keypoint transfer accuracy 3D Chamfer distance F-score mIoU training time (A100 GPU hours)
Baselines: LASR ViSER BANMo Hi-LASSIE RigNet Skeletor MagicPony
Builds on: LASR ViSER BANMo Hi-LASSIE ARAP
Closest papers by meaning
- Learning Implicit Representation for Reconstructing Articulated Objects (ICLR 2024 · similarity 0.88)
- REACTO: Reconstructing Articulated Objects from a Single Video (CVPR 2024 · similarity 0.88)
- SAMoR: Motion Modelling for Articulated Objects of Any Skeleton and Topology (NeurIPS 2026 · similarity 0.87)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.87)
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (ICLR 2025 · similarity 0.87)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.