Research map / 3D vision / Face and shape
Physics-guided Shape-from-Template: Monocular Video Perception through Neural Surrogate Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Shape-from-Template method for cloth that uses a pre-trained physics-based neural surrogate cloth simulator, nvdiffrast differentiable rendering and gradient-based optimization to recover shape and stiffness parameters from monocular video, running 400-500x faster than φ-SfT.
Problem
Monocular shape-from-template reconstruction of cloth is either unphysical/noisy or slow to optimize when using physics simulation as regularization.
Approach
Replace the classical cloth simulator in a differentiable physics-based SfT pipeline with a self-supervised neural surrogate (U-Net over a 32x32 vertex grid trained with a physics energy loss), enabling fast backpropagation-through-time optimization of shape, stiffness, external forces and uv-map.
Key results
- Runtime reduced from about 17:45-24:12 h per scene (phi-SfT) to 2.5-3 min per scene, a speedup of roughly 380-585x.
- Chamfer distance comparable to phi-SfT: ours better on R5 and R7 (11.7 vs 14.4, 6.9 vs 9.1), phi-SfT better on 4 of 6 scenes.
- Ablations show removing the uv-map optimization or turbulent external forces causes large quality drops (e.g., R3 13.3 and 21.0 vs 12.5 full).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add silhouette loss (ablation) | Ours · ϕ-SfT R3 | symmetric Chamfer distance ×104: 13.8 -> 12.5 | Helped |
| Add regularization for external forces (ablation) | Ours · ϕ-SfT R4 | symmetric Chamfer distance ×104: 14.8 -> 14.5 | No clear effect |
| Add turbulent external forces (ablation) | Ours · ϕ-SfT R3 | symmetric Chamfer distance ×104: 21.0 -> 12.5 | Helped |
| Add uv-map optimization (ablation) | Ours · ϕ-SfT R4 | symmetric Chamfer distance ×104: 24.1 -> 14.5 | Helped |
| Use successive optimization rather than optimizing all frames from the beginning (ablation) | Ours · ϕ-SfT R4 | symmetric Chamfer distance ×104: 19.0 -> 14.5 | Helped |
Surprising findings
- Removing the regularization term on external forces barely changes reconstruction quality within the given iterations.
- Overfitting after about 400 epochs worsens Chamfer distance, although the optimization remains stable.
Limitations
- Cannot capture fine wrinkles or high-curvature details at 32x32 grid resolution.
- uv-map optimization can cause texture distortions and smearing of straight lines or patterns.
- Quantitative accuracy is comparable to, not better than, phi-SfT on most scenes.
Experimental setup
Datasets: phi-SfT dataset
Benchmarks: phi-SfT dataset (scenes R3, R4, R5, R7, R8, R9)
Metrics: symmetric Chamfer distance optimization runtime (minutes) speedup factor
Baselines: phi-SfT
Builds on: phi-SfT nvdiffrast Shape-from-Template Neural Cloth Simulation (self-supervised physics loss)
Closest papers by meaning
- Thin-Shell-SfT: Fine-Grained Monocular Non-rigid 3D Surface Tracking with Neural Deformation Fields (CVPR 2025 · similarity 0.87)
- Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulation (CVPR 2025 · similarity 0.87)
- MonoPhysics: Estimating Geometry, Appearance, and Physical Parameters from Monocular Videos (NeurIPS 2026 · similarity 0.86)
- MorpheuS: Neural Dynamic 360deg Surface Reconstruction from Monocular RGB-D Video (CVPR 2024 · similarity 0.86)
- NeuralClothSim: Neural Deformation Fields Meet the Thin Shell Theory (NeurIPS 2024 · similarity 0.86)
- MultiGO: Towards Multi-level Geometry Learning for Monocular 3D Textured Human Reconstruction (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.