Research map / Image generation / Text-to-image diffusion
DreamFit: Garment-Centric Human Generation via a Lightweight Anything-Dressing Encoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamFit, a garment-centric human generation framework that turns the frozen SD1.5 denoising UNet into a lightweight LoRA-based reference encoder with adaptive attention, plus LMM-based prompt rewriting at inference.
Problem
Garment-centric human generation from text and garment image prompts faces a trade-off: lightweight adapters produce inconsistent textures, while full finetuning (e.g., ReferenceNet copies) is resource-heavy and erodes pretrained priors; training and inference text prompts also differ in detail.
Approach
Use LoRA layers switched on inside the frozen denoising UNet to act as a feature extractor for noise-free garment images (the "Anything-Dressing Encoder"), inject its features through an adaptive attention that adds a trainable-key/value cross-attention branch to self-attention, and rewrite user prompts with an LMM at inference to match the detailed training captions.
Key results
- On the open benchmark DreamFit reports CLIP-T 0.328, CLIP-I 0.740, AS 5.414, HAS 0.856, the best among compared methods, with 83.4M trainable parameters versus 859M-875M for full-finetune baselines.
- On the internal benchmark DreamFit reports CLIP-T 0.334, CLIP-I 0.687, HAS 0.920, outperforming IMAGDress (HAS 0.840) and StableGarment (HAS 0.683).
- In the prompt ablation, LMM-rewritten prompts reach CLIP-I 0.687 and a human study preference of 0.87 versus 0.10 for simple text and 0.03 for fixed text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DreamFit instead of the strongest baseline | Garment-centric human generation on Open Benchmark · Open Benchmark | CLIP-T: 0.300 -> 0.328 | Helped |
| Use DreamFit instead of the strongest baseline | Garment-centric human generation on Open Benchmark · Open Benchmark | CLIP-I: 0.720 -> 0.740 | Helped |
| Use DreamFit instead of the strongest baseline | Garment-centric human generation on Open Benchmark · Open Benchmark | AS: 5.007 -> 5.414 | Helped |
| Use DreamFit instead of the strongest baseline | Garment-centric human generation on Open Benchmark · Open Benchmark | HAS: 0.783 -> 0.856 | Helped |
| Use DreamFit instead of the strongest baseline | Garment-centric human generation on Internal Benchmark · Internal Benchmark | CLIP-T: 0.320 -> 0.334 | Helped |
| Use DreamFit instead of the strongest baseline | Garment-centric human generation on Internal Benchmark · Internal Benchmark | CLIP-I: 0.632 -> 0.687 | Helped |
| Use DreamFit instead of the strongest baseline | Garment-centric human generation on Internal Benchmark · Internal Benchmark | AS: 5.462 -> 5.516 | Helped |
| Use DreamFit instead of the strongest baseline | Garment-centric human generation on Internal Benchmark · Internal Benchmark | HAS: 0.840 -> 0.920 | Helped |
| Add LoRA layers and adaptive-attention adapters instead of fine-tuning the entire encoder (ablation) | DreamFit · Ablation evaluation | CLIP-I: 0.622 -> 0.687 | Helped |
| Add adaptive-attention adapters to the LoRA-only configuration (ablation) | DreamFit · Ablation evaluation | AS: 5.250 -> 5.516 | Helped |
| Add LoRA layers to the adapter-only configuration (ablation) | DreamFit · Ablation evaluation | CLIP-T: 0.310 -> 0.334 | Helped |
| Rewrite prompts with an LMM instead of using fixed text (ablation) | DreamFit · Text prompt ablation | CLIP-I: 0.588 -> 0.687 | Helped |
Surprising findings
- Full finetuning of the encoder achieved the highest aesthetic score (5.534) but lower text and texture consistency (CLIP-T 0.322, CLIP-I 0.622) than the LoRA-plus-adapter combination.
- MagicClothing scored lower on HAS than IP-Adapter on the open benchmark (0.443 vs 0.568), despite higher CLIP-T.
Limitations
- Runs out of memory when training the FLUX version at 1024x768 even on 8 A100 80G GPUs.
- Evaluation relies on CLIP metrics and GPT-4o scoring, with a human study on only a subset of prompt configurations.
- Internal benchmark is small (200 images) and collected from the internet, limiting reproducibility.
Experimental setup
Datasets: VITON-HD DressCode
Benchmarks: Open benchmark (200 garments from VITON-HD and DressCode) Internal benchmark (200 internet reference images)
Metrics: CLIP-T CLIP-I Aesthetic score Human-Aligned Score human preference study
Baselines: IP-Adapter StableGarment MagicClothing IMAGDress
Builds on: Stable Diffusion LoRA IP-Adapter ControlNet
Closest papers by meaning
- AnyDressing: Customizable Multi-Garment Virtual Dressing via Latent Diffusion Models (CVPR 2025 · similarity 0.90)
- Controllable Human Image Generation with Personalized Multi-Garments (CVPR 2025 · similarity 0.89)
- FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models (NeurIPS 2024 · similarity 0.88)
- Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On (CVPR 2024 · similarity 0.87)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.87)
- ChatGarment: Garment Estimation, Generation and Editing via Large Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.