Research map / 3D vision / Face and shape
GaPT-DAR: Category-level Garments Pose Tracking via Integrated 2D Deformation and 3D Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GaPT-DAR, a category-level garment pose tracker that projects 3D point clouds to 2D via a voting-based plane, deforms them with TPS in 2D, and reconstructs depth back to 3D, evaluated on VR-Folding.
Problem
Category-level garment pose tracking from partial point cloud sequences is hard because prior methods predict per-point canonical (NOCS) correspondences purely in 3D, which poorly captures garment warping and geometric symmetry.
Approach
Reformulate garment tracking as 2D warping: learn an optimal 3D-to-2D projection plane via voting-based offsets and heatmaps, warp the 2D point sets with a TPS deformation predicted from features, and recover depth per pixel with a depth-reconstruction network trained with Log-L1 and depth-gradient losses.
Key results
- On VR-Folding Shirt Folding, A3cm improves from 29.8% (GarmentTracking) to 32.3%, and Shirt Flattening Dnocs drops from 0.105 to 0.098.
- Pants Folding Dnocs reaches 0.031 vs 0.039 for GarmentTracking, with A3cm 49.1% vs 47.3%.
- Under noise level 3x and frame keep ratio 1/8, the method keeps lower Dnocs than GarmentTracking across tests on Pants Folding and Flattening.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use voting-based rather than SVD-based projection (ablation) | GaPT-DAR · VR-Folding Pants Folding | Dchamf: 1.53 -> 0.99 | Helped |
| Use voting-based rather than SVD-based projection (ablation) | GaPT-DAR · VR-Folding Pants Flattening | Dnocs: 0.247 -> 0.101 | Helped |
| Use voting-based rather than fixed xOz plane projection (ablation) | GaPT-DAR · VR-Folding Pants Folding | Dchamf: 1.73 -> 0.99 | Helped |
| Use voting-based rather than fixed xOz plane projection (ablation) | GaPT-DAR · VR-Folding Pants Flattening | Dnocs: 0.287 -> 0.101 | Helped |
| Use integrated 3D-2D deformation learning instead of direct 3D deformation prediction (ablation) | GaPT-DAR · VR-Folding dataset | performance | Helped |
| Increase input perturbation noise (ablation) | GaPT-DAR · VR-Folding dataset | Dnocs: 0.098 -> 0.109; 0.098 -> 0.151; 0.098 -> 0.161 | Hurt |
| Reduce frame keep ratio (ablation) | GaPT-DAR · VR-Folding dataset | Dnocs: 0.098 -> 0.106; 0.098 -> 0.145; 0.098 -> 0.164; 0.098 -> 0.178 | Hurt |
Surprising findings
- Direct 3D regression of complete NOCS coordinates performs notably worse, especially on flattening, than the 3D-2D warping pipeline.
- The paper reports that the xOz plane, which is a fixed non-parametric projection, performs worse than a learned SVD-based projection.
Limitations
- Performance degrades under severe self-occlusion where occluded parts are imperceptible.
- Requires large amounts of carefully labeled high-quality data and aligned 3D meshes, which are costly.
- Sim-to-real evaluation is only qualitative on real-world data.
Experimental setup
Datasets: VR-Folding CLOTH3D
Benchmarks: VR-Folding
Metrics: A3cm A5cm A10cm Dcorr Dchamf Dnocs
Baselines: GarmentNets GarmentTracking SVD-based projection xOz plane projection
Builds on: GarmentTracking GarmentNets NOCS ResUNet3D
Closest papers by meaning
- GarmentTracking: Category-Level Garment Pose Tracking (CVPR 2023 · similarity 0.92)
- Garment Recovery with Shape and Deformation Priors (CVPR 2024 · similarity 0.87)
- Learning 3D Garment Animation from Trajectories of A Piece of Cloth (NeurIPS 2024 · similarity 0.87)
- Reconstruction of Manipulated Garment with Guided Deformation Prior (NeurIPS 2024 · similarity 0.87)
- DressWild: Feed-Forward Pose-Agnostic Garment Sewing Pattern Generation from In-the-Wild Images (NeurIPS 2026 · similarity 0.86)
- VS: Reconstructing Clothed 3D Human from Single Image via Vertex Shift (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.