Research map / Reinforcement learning / Visual and representation RL
DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DexGarmentLab, a bimanual dexterous garment manipulation environment in Isaac Sim with an automated single-demo data collection pipeline and a hierarchical policy (HALO) combining affordance-based grasp point localization (GAM) and a structure-aware diffusion policy (SADP).
Problem
Dexterous, especially bimanual, garment manipulation lacks realistic simulation, efficient demonstration collection (teleoperation or RL is labor-intensive), and policies that generalize across garment shapes and deformations.
Approach
Combine a category-level garment affordance model (dense correspondence via UniGarmentManip) that transfers demo grasp points to new garments with a structure-aware diffusion policy conditioned on those affordance points, plus a PBD simulation with adhesion/friction/particle-scale parameters so dexterous hands can grasp garments physically, and automated data generation from a single expert demo.
Key results
- HALO achieves 0.82±0.06 on Fling Dress and 0.92±0.04 on Hang Tops in simulation, outperforming DP (0.59) and DP3 (0.51) on Fling Dress.
- In real-world evaluation over 15 trials per task, HALO reaches 13/15 on Fold Tops and Hang Tops, 11/15 on Wear Scarf, and 14/15 on Wear Hat, versus at most 10/15 for DP.
- Sim-to-real with only simulation data drops Hang Trousers to 8/15 and Wear Hat to 9/15; adding 15 real-world demos raises both to 13/15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GAM for garment affordance-point localization (ablation) | HALO · Simulation garment manipulation tasks | success rate: Fling Dress: 0.66 ± 0.02 -> 0.82 ± 0.06; Hang Tops: 0.64 ± 0.02 -> 0.92 ± 0.04 | Helped |
| Add SADP for structure-aware trajectory generation instead of DP3 (ablation) | HALO · Simulation garment manipulation tasks | success rate: Fling Dress: 0.68 ± 0.09 -> 0.82 ± 0.06; Hang Tops: 0.70 ± 0.08 -> 0.92 ± 0.04 | Helped |
| Use HALO rather than DP or DP3 | garment manipulation policies · Simulation garment manipulation tasks | success rate: Fling Dress: 0.59 ± 0.05 -> 0.82 ± 0.06; Wear Bowlhat: 0.55 ± 0.04 -> 0.72 ± 0.04 | Helped |
| Use HALO rather than DP or DP3 | garment manipulation policies · Real-world garment manipulation tasks | success rate: Fold Tops: 9 / 15 -> 13 / 15; Wear Hat: 10 / 15 -> 14 / 15 | Helped |
| Add 15 real-world training examples to simulation training data | HALO · Hang Trousers (real-world evaluation) | success rate: 8 / 15 (53.3\%) -> 13 / 15 (86.7\%) | Helped |
| Add 15 real-world training examples to simulation training data | HALO · Wear Hat (real-world evaluation) | success rate: 9 / 15 (60.0\%) -> 13 / 15 (86.7\%) | Helped |
| Use automated collection rather than teleoperation data | HALO · Hang Tops (simulation) | success rate: 0.88±0.03 -> 0.92±0.04 | No clear effect |
Surprising findings
- Eureka (RL+VLM) reached only 0.16 and 0.08 success on Hang Coat and Wear Bowlhat, and produced no valid results on the other tasks.
- Policies trained on automatically collected demonstrations perform comparably to those trained on teleoperation data (0.92 vs 0.88 on Hang Tops in simulation).
Limitations
- PBD and FEM simulation methods only approximate real garment behavior, with jitter and penetration artifacts and FEM garments reverting to original shape.
- RL-based baselines were not validly evaluated because Isaac Sim lacks robust parallelization for deformable objects.
- Real-world experiments cover only a few tasks, and sim-to-real transfer needed real-world data to work well.
Experimental setup
Datasets: ClothesNet
Benchmarks: DexGarmentLab simulation tasks (Fling, Fold, Hang, Wear, Store) real-world Fold Tops, Hang Tops, Wear Scarf, Wear Hat
Metrics: success rate
Baselines: Diffusion Policy DP3 ACT pi0 RDT Eureka
Builds on: Diffusion Policy UniGarmentManip 3D Diffusion Policy PointNet++
Closest papers by meaning
- GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation (NeurIPS 2024 · similarity 0.93)
- GarmentPile: Point-Level Visual Affordance Guided Retrieval and Adaptation for Cluttered Garments Manipulation (CVPR 2025 · similarity 0.90)
- Learning Efficient Robotic Garment Manipulation with Standardization (ICML 2025 · similarity 0.89)
- UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence (CVPR 2024 · similarity 0.89)
- Learning 3D Garment Animation from Trajectories of A Piece of Cloth (NeurIPS 2024 · similarity 0.87)
- DressWild: Feed-Forward Pose-Agnostic Garment Sewing Pattern Generation from In-the-Wild Images (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.