Research map / Image generation / Text-to-image diffusion
MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a ~1M human-centric image dataset with close-up face and hand subsets, then fine-tunes SD v1.5 and combines low-rank face and hand experts via soft local and global gating to improve human-centric text-to-image generation.
Problem
Text-to-image diffusion models produce unnatural faces and hands in human-centric generation, due to insufficient high-quality human-centric training priors and the high variability of faces and hands.
Approach
Treats LoRA-style low-rank modules trained on close-up face and hand datasets as experts in a soft mixture-of-experts, with learned sigmoid local (per-token) and global (per-layer) gates added residually to frozen base UNet linear layers; the observation that a low-rank module at a proper scale refines its corresponding body part motivates this.
Key results
- On COCO Human Prompts, MoLE (SD v1.5) reaches 20.27 HPS and 33.75 ImageReward vs. 19.91 and 28.34 for SD v1.5 base.
- MoLE on SDXL substantially outperforms SDXL in the generalization experiments; on SD v2.1 HPS rises from 20.17 to 20.45.
- In a user study with 50 participants, 58% preferred MoLE over HyperHuman and 48% over HanDiffuser on hand quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MoLE face and hand experts with soft assignment | SD v2.1 · COCO Human Prompts | HPS (%): $20.17 \pm 0.08$ -> $20.45 \pm 0.09$ | Helped |
| add MoLE face and hand experts with soft assignment | SD v2.1 · COCO Human Prompts | ImageReward: $58.36 \pm 0.51 $ -> $62.08 \pm 0.86$ | Helped |
| add MoLE face and hand experts with soft assignment | PixArt-XL-2-512×512 · COCO Human Prompts | HPS (%): $21.33 \pm 0.08$ -> $21.79 \pm 0.03$ | Helped |
| fine-tune on the human-centric dataset (ablation) | SD v1.5 · COCO Human Prompts | HPS (%): $19.91 \pm 0.09$ -> $20.16 \pm 0.09$ | Helped |
| fine-tune on the human-centric dataset (ablation) | SD v1.5 · COCO Human Prompts | ImageReward: $28.34 \pm 1.40$ -> $31.01 \pm 1.75$ | Helped |
| add the face and hand low-rank experts without Stage 3 gating (ablation) | SD v1.5 fine-tuned on the human-centric dataset · COCO Human Prompts | HPS (%): $20.16 \pm 0.09$ -> $19.94 \pm 0.07$ | Hurt |
| add the face and hand low-rank experts without Stage 3 gating (ablation) | SD v1.5 fine-tuned on the human-centric dataset · COCO Human Prompts | ImageReward: $31.01 \pm 1.75$ -> $25.66 \pm 2.72$ | Hurt |
| add soft assignment gating for the face and hand experts (ablation) | SD v1.5 with Stage 1 and Stage 2 · COCO Human Prompts | HPS (%): $19.94 \pm 0.07$ -> $20.27 \pm 0.07$ | Helped |
| add soft assignment gating for the face and hand experts (ablation) | SD v1.5 with Stage 1 and Stage 2 · COCO Human Prompts | ImageReward: $25.66 \pm 2.72$ -> $33.75 \pm 1.49$ | Helped |
| use local assignment only (ablation) | MoLE on SD v1.5 · COCO Human Prompts | HPS (%): $19.91 \pm 0.09$ -> $20.19 \pm 0.04$ | Helped |
| use global assignment only (ablation) | MoLE on SD v1.5 · COCO Human Prompts | HPS (%): $19.91 \pm 0.09$ -> $20.20 \pm 0.02$ | Helped |
| combine local and global assignment (ablation) | MoLE on SD v1.5 · COCO Human Prompts | ImageReward: $31.98 \pm 2.41 $ -> $33.75 \pm 1.49$; $32.20 \pm 0.86$ -> $33.75 \pm 1.49$ | Helped |
Surprising findings
- Adding the two close-up experts without gating (Stage 2) lowered HPS and ImageReward relative to fine-tuned Stage 1, which the authors attribute to experts mimicking close-up data distributions.
- Only about 25% of MoLE outputs were high quality on identical prompts, though this is still above SDXL's 10%.
Limitations
- Struggles with multi-person scenes, since the dataset is mostly single-person images.
- Only about 25% of outputs are high quality on identical prompts, below practical standards.
- Dataset demographic bias is inherited from source websites; race balance analysis shows skew toward White individuals.
Experimental setup
Datasets: Celeb-HQ FFHQ 11k Hands LAION2b-en COCO Caption DiffusionDB CosmicMan
Benchmarks: COCO Human Prompts DiffusionDB Human Prompts
Metrics: HPS ImageReward CLIP-T FID Aesthetic Score
Baselines: SD v1.5 SDXL VQ-Diffusion Versatile Diffusion HanDiffuser HyperHuman PixArt-alpha SD v2.1
Builds on: LoRA Soft MoE Stable Diffusion v1.5 Min-SNR weighting Lion optimizer
Closest papers by meaning
- Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation (CVPR 2024 · similarity 0.89)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.87)
- Limitations of Face Image Generation (AAAI 2024 · similarity 0.87)
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.87)
- CosmicMan: A Text-to-Image Foundation Model for Humans (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.