Research map / Image generation / Text-to-image diffusion
CosmicMan: A Text-to-Image Foundation Model for Humans
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CosmicMan-HQ (6M human images with dense attribute annotations via a human-in-the-loop "Annotate Anyone" pipeline) and fine-tunes SD/SDXL with the Daring framework and HOLA attention loss to improve human text-to-image generation.
Problem
General-purpose text-to-image models produce low-quality humans with structural errors and poor alignment to dense, detailed human descriptions; there is no human-specialized foundation model and no scalable pipeline for high-quality human data.
Approach
A human-AI cooperative data flywheel (Annotate Anyone) produces large, precisely annotated human data, and Daring decomposes captions into body/outfit groups aligned with human parsing maps, adding a HOLA cross-attention loss to refocus attention without extra modules.
Key results
- CosmicMan-SDXL reaches FID 35.42 and Accall 83.6 versus SDXL's 48.61 and 78.1 and DALLE3's 77.8 on the 2048-image human test set.
- Pairwise user study prefers CosmicMan over DeepFloyd-IF, SDXL, DALLE3 and MidJourney in image quality at 93.06%, 82.93%, 78.13% and 70.43%.
- Training on CosmicMan-HQ (6M) lowers FID from 48.09 to 37.57 over the SD baseline; adding HOLA brings it to 36.78 with Accall 81.2 for CosmicMan-SD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SDXL for human image generation | SDXL · Human image generation test set | FID: 48.61 -> 35.42 | Helped |
| Fine-tune SDXL for human image generation | SDXL · Human image generation test set | Accall: 78.1 -> 83.6 | Helped |
| Use CosmicMan-SDXL instead of SDXL in T2I-Adapter | T2I-Adapter + SDXL · 2D human editing | Accall: 76.6 -> 82.9 | Helped |
| Use CosmicMan-SD instead of SD in Magic123 | Magic123 + SD · 3D human reconstruction | Accall: 67.6 -> 70.8 | Helped |
| Increase training data from 1M to 6M images (ablation) | CosmicMan trained with Annotate Anyone captions · Human image generation test set | FID: 40.08 -> 37.57 | Helped |
| Increase training data from 1M to 6M images (ablation) | CosmicMan trained with Annotate Anyone captions · Human image generation test set | Accall: 78.8 -> 79.7 | Helped |
| Use Annotate Anyone captions instead of web alternative text (ablation) | 1M-image CosmicMan training · Human image generation test set | FID: 47.65 -> 40.08 | Helped |
| Use Annotate Anyone captions instead of pretrained InstructBLIP captions (ablation) | 1M-image CosmicMan training · Human image generation test set | Accall: 75.6 -> 78.8 | Helped |
| Fine-tune on CosmicMan-HQ (ablation) | SD baseline · Human image generation test set | FID: 48.09 -> 37.57 | Helped |
| Add HOLA loss (ablation) | CosmicMan trained on CosmicMan-HQ · Human image generation test set | Accall: 79.7 -> 81.2 | Helped |
Surprising findings
- CosmicMan-SDXL has a lower CLIPScore than the SDXL and DALLE3 baselines, which the authors attribute to CLIPScore's poor fine-grained sensitivity.
Limitations
- CLIPScore is low for the proposed models and the custom Accall metric may not fully capture fine-grained alignment.
- The user study uses only 100 prompts, and the test set has 2048 images with manually annotated prompts.
Experimental setup
Datasets: CosmicMan-HQ 1.0 LAION-5B SHHQ DeepFashion HumanSD LAION-Human
Benchmarks: 2048-image human test set with manual prompts
Metrics: FID HPSv2 CLIPScore Accobj Acctex Accshape Accall user preference
Baselines: SD 1.5 SD 2.0 SDXL DeepFloyd-IF DALLE2 DALLE3 MidJourney HumanSD LAION-5B
Builds on: Stable Diffusion SDXL InstructBLIP SCHP Magic123
Closest papers by meaning
- Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation (CVPR 2024 · similarity 0.88)
- HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion (ICLR 2024 · similarity 0.87)
- MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts (NeurIPS 2024 · similarity 0.87)
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- L-Man: A Large Multi-modal Model Unifying Human-centric Tasks (AAAI 2025 · similarity 0.87)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.