Research map / Vision-language and multimodal / Multimodal LLMs
ChatGarment: Garment Estimation, Generation and Editing via Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes LLaVA-1.5-7B to output GarmentCode JSON garment configurations from images or text, enabling sewing-pattern estimation, generation, and editing via dialogue.
Problem
Estimating, generating, and editing 3D garments as sewing patterns from images or text is hard because prior methods lack robustness on in-the-wild inputs, have limited training data, and do not support interactive editing.
Approach
Finetune a vision-language model to emit a JSON garment description (GarmentCode) with textual tokens plus continuous numeric values decoded by an MLP projection from the <ENDS> token embedding; pair with a refined, more compact GarmentCodeRC and an automated GPT-4o-labeled data pipeline.
Key results
- On Dress4D (target pose), CD 3.12 and F-Score 0.75 vs GarmentRecovery CD 9.69 and F-Score 0.55; zero stitching failures.
- On CLoSE, F-Score 0.790 (best) with CD 2.94, slightly behind GarmentRecovery's CD 2.39.
- Garment editing CD 2.51 and F-Score 0.893 vs DressCode CD 10.97 and F-Score 0.750; text-based generation CLIP score 23.7 vs DressCode 22.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the numeric projection layer instead of directly generating numeric values | LLaVA* image-based reconstruction · Dress4D Target-Pose CD | CD: 5.29 -> 3.12 | Helped |
| Combine image input with a generated text description using the two-step CoT procedure | LLaVA image-based reconstruction · Dress4D Target-Pose CD | CD: 4.32 -> 3.12 | Helped |
| Add part-level garment descriptions to training data (ablation) | ChatGarment image reconstruction · CLoSE | CD: 4.06 -> 2.94 | Helped |
| Add part-level garment descriptions to training data (ablation) | ChatGarment image reconstruction · Dress4D | F-Score: 0.79 -> 0.78 | Mixed |
| Use LLaVA-13B instead of LLaVA-7B (ablation) | ChatGarment image reconstruction · Dress4D | CD: 3.06 -> 3.73 | Hurt |
| Use LLaVA-13B instead of LLaVA-7B (ablation) | ChatGarment image reconstruction · CLoSE | CD: 2.94 -> 2.54 | Hurt |
| Use a JSON configuration with omitted irrelevant settings and normalized values | GarmentCode · VLM training token count | average language token count: 900 to 350 | Helped |
Surprising findings
- Direct edge and stitch prediction (SewFormer, DressCode) did not improve reconstruction accuracy and increased invalid patterns on out-of-distribution data.
- Model generalizes zero-shot to sketch inputs and multi-turn dialogue despite training only on single-turn data.
Limitations
- Precise garment editing may unintentionally alter other parts, e.g., changing skirt length affects other regions.
- GarmentRecovery depends on 2D segmentation and does not support whole-body dresses; SewFormer fails on detection or stitching.
- Synthetic training data generated from GarmentCode and GPT-4o labels may limit real-world coverage.
Experimental setup
Datasets: GarmentCodeData BEDLAM SHHQ LLaVA-v1.5-mix665k CLoSE Dress4D
Benchmarks: Dress4D CLoSE ChatGarment editing eval set (135 pairs) text-based generation prompts (DressCode protocol)
Metrics: chamfer distance F-Score stitching failure rate CLIP score
Baselines: SewFormer GarmentRecovery DressCode GPT-4o LLaVA* LLaVA-T LLaVA-I
Builds on: GarmentCode LLaVA ChatPose DressCode GarmentCodeData
Closest papers by meaning
- AIpparel: A Multimodal Foundation Model for Digital Garments (CVPR 2025 · similarity 0.90)
- DressWild: Feed-Forward Pose-Agnostic Garment Sewing Pattern Generation from In-the-Wild Images (NeurIPS 2026 · similarity 0.90)
- Design2Cloth: 3D Cloth Generation from 2D Masks (CVPR 2024 · similarity 0.88)
- IMAGDressing-v1: Customizable Virtual Dressing (AAAI 2025 · similarity 0.88)
- AnyDressing: Customizable Multi-Garment Virtual Dressing via Latent Diffusion Models (CVPR 2025 · similarity 0.88)
- FashionTailor: Controllable Clothing Editing for Human Images with Appearance Preserving (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.