Research map / Point clouds / Shape and surface completion
Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Motion2VecSets is a 4D latent-set diffusion model that reconstructs non-rigid surface sequences from sparse, noisy, or partial point clouds by denoising shape and deformation vector sets with interleaved spatio-temporal attention.
Problem
Dynamic surface reconstruction and tracking of non-rigid objects from ambiguous, sparse, noisy, or partial point cloud sequences, where deterministic feed-forward/neural-field methods struggle with generalization to unseen identities and motions.
Approach
Represent a 4D sequence as a shape latent set for the reference frame plus per-frame deformation latent sets, and diffuse these sets jointly (synchronized denoising with a shared noise level) using interleaved space self-attention, condition cross-attention, and time self-attention blocks.
Key results
- On DT4D-A sparse-noisy reconstruction (unseen individual), Motion2VecSets reaches 83.7% IoU and 0.058 CD vs CaDeX's 64.7% IoU and 0.127 CD.
- On D-FAUST monocular depth completion, Motion2VecSets achieves 83.8% IoU, 0.054 CD, 0.111 Corr on unseen motion, outperforming CaDeX's 80.7%, 0.074, 0.123.
- Ablation on D-FAUST shows removing time attention or using a single global code (M=1) lowers IoU on unseen motion from 83.8% to 81.2% and 68.5%, respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Motion2VecSets instead of CaDeX | 4D shape reconstruction · DT4D-A unseen individual, sparse/noisy point clouds | IoU: 64.7% -> 83.7% | Helped |
| use Motion2VecSets instead of CaDeX | 4D shape reconstruction · D-FAUST unseen motion, sparse/noisy point clouds | IoU: 89.1% -> 90.7% | Helped |
| use Motion2VecSets instead of CaDeX | 4D shape reconstruction · D-FAUST unseen individual, sparse/noisy point clouds | IoU: 80.7% -> 83.7% | Helped |
| use Motion2VecSets instead of CaDeX | 4D shape completion · DT4D-A unseen individual, monocular noisy depth scans | IoU: 59.2% -> 66.3% | Helped |
| replace diffusion with one-step regression (ablation) | Motion2VecSets · D-FAUST unseen motion, monocular noisy depth scans | IoU: 71.1% -> 83.8% | Helped |
| use latent vector sets instead of one global latent code (ablation) | Motion2VecSets · D-FAUST unseen motion, monocular noisy depth scans | IoU: 68.5% -> 83.8% | Helped |
| use C = 32 latent channels instead of C = 8 (ablation) | Motion2VecSets · D-FAUST unseen motion, monocular noisy depth scans | IoU: 78.9% -> 83.8% | Helped |
| use C = 32 latent channels instead of C = 16 (ablation) | Motion2VecSets · D-FAUST unseen motion, monocular noisy depth scans | IoU: 78.0% -> 83.8% | Helped |
| add time self-attention (ablation) | synchronized deformation latent set diffusion · D-FAUST unseen motion, monocular noisy depth scans | IoU: 81.2% -> 83.8% | Helped |
Limitations
- Evaluated only on two datasets with simulated sparse, noisy, and partial inputs.
- Ambiguity handling is shown qualitatively; diversity of samples is not quantified.
Experimental setup
Datasets: Dynamic FAUST (D-FAUST) DeformingThings4D-Animals (DT4D-A)
Benchmarks: D-FAUST DT4D-A
Metrics: IoU Chamfer distance correspondence L2 error
Baselines: OFlow LPDC CaDeX
Builds on: 3DShape2VecSet EDM latent diffusion LPDC CaDeX
Closest papers by meaning
- Adaptive 3D Reconstruction via Diffusion Priors and Forward Curvature-Matching Likelihood Updates (NeurIPS 2025 · similarity 0.88)
- 4Deform: Neural Surface Deformation for Robust Shape Interpolation (CVPR 2025 · similarity 0.87)
- 4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians (CVPR 2025 · similarity 0.87)
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.87)
- Controlling Space and Time with Diffusion Models (ICLR 2025 · similarity 0.87)
- NeuroGauss4D-PCI: 4D Neural Fields and Gaussian Deformation Fields for Point Cloud Interpolation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.