Research map / Reinforcement learning / Visual and representation RL

GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation

NeurIPS 2024 · Reinforcement learning › Visual and representation RL · approach: Dexterous manipulation transfer · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GarmentLab, a GPU-parallel simulation environment, asset library, 20-task benchmark, and real-world deformable-object benchmark with sim-to-real methods, evaluated on vision-based and RL garment manipulation baselines.

Problem

Garment and deformable-object manipulation lacks realistic, diverse, multi-physics simulation benchmarks and real-world benchmarks, and existing sim environments are CPU-based, single-physics, or lack robots and sim-to-real support.

Approach

A unified GPU-parallel simulator on Isaac Sim combining PBD and FEM with fluids, rigid/articulated objects, humans and multiple robots, plus a 20-task benchmark, a real-world garment benchmark, and sim-to-real components (keypoint embedding alignment, noise augmentation, point cloud alignment, teleoperation, MoveIt).

Key results

• In simulation, UGM scores ~61.5/62.1/59.8 on folding large garments, while DIFT reaches 66.4/63.2 on placing small pieces and 64.7/61.2 on hat hanging.
• RL-State and RL-Vision PPO perform poorly across tasks (e.g., 6.7–14.8 on large-piece tasks), attributed to abnormal trajectories and high-dimensional visual input.
• Real-world T-shirt folding: UGM 10/15 vs DIFT 8/15 and Affordance 6/15; removing point cloud alignment drops UGM to 2/15.

Measured effects reported in the paper

Surprising findings

• DIFT, a 2D correspondence model not designed for garments, outperforms UGM on small-piece tasks due to robustness to rotation.
• Vision-based PPO performs worse than state-based PPO, and both underperform predefined-trajectory visual methods on garment tasks.

Limitations

• Authors state code will be open-sourced 'as soon as possible', so availability is not yet confirmed.
• RL baselines perform poorly, showing current methods struggle with long-horizon and high-dimensional garment tasks.
• Real-world evaluation covers only two tasks with 15 trials each, and only vision-based methods were tested in the real world.

Experimental setup

Datasets: ClothesNet ShapeNet PartNet PartNet-Mobility YCB Omniverse HumanModel ActorCore

Benchmarks: GarmentLab Benchmark (20 tasks) GarmentLab real-world benchmark

Metrics: success rate IoU for folding coverage/vertex-distance for unfolding

Baselines: UniGarmentManip DIFT Affordance RL-State (PPO) RL-Vision (PPO)

Builds on: Isaac Sim UniGarmentManip DIFT ClothesNet ROS

Closest papers by meaning

• DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy (NeurIPS 2025 · similarity 0.93)
• Learning Efficient Robotic Garment Manipulation with Standardization (ICML 2025 · similarity 0.91)
• GarmentPile: Point-Level Visual Affordance Guided Retrieval and Adaptation for Cluttered Garments Manipulation (CVPR 2025 · similarity 0.90)
• UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence (CVPR 2024 · similarity 0.90)
• Learning 3D Garment Animation from Trajectories of A Piece of Cloth (NeurIPS 2024 · similarity 0.89)
• DressWild: Feed-Forward Pose-Agnostic Garment Sewing Pattern Generation from In-the-Wild Images (NeurIPS 2026 · similarity 0.87)