Research map / Image generation / Text-to-image diffusion
SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SELMA, which generates skill-specific prompts with an LLM, self-generates images with the T2I model, fine-tunes per-skill LoRA experts, and merges them by weight averaging to improve text faithfulness and human preference of Stable Diffusion models.
Problem
Text-to-image diffusion models often fail to faithfully render details in prompts (object counts, spatial relations, text rendering, missing objects), and existing fixes rely on costly human annotation, reward models, or layouts.
Approach
Use an LLM with in-context seed prompts to auto-generate skill-specific prompt datasets (ROUGE-L filtered), have the T2I model itself generate one image per prompt, fine-tune a separate LoRA expert per dataset, and merge the LoRA A and B matrices by uniform averaging at inference so one model covers all skills without routing.
Key results
- On SD v2, SELMA raises DSG accuracy from 70.3 to 77.7 and TIFA from 79.2 to 83.2, outperforming single-LoRA and MoE-LoRA variants.
- On SDXL, SELMA improves DSG from 73.3 to 80.2 and TIFA from 83.5 to 85.6, and in human evaluation SDXL+SELMA wins 241 vs 114 losses on 200 DSG prompts.
- Fine-tuning SDXL on images generated by the weaker SD v2 reaches DSG 81.3, above SDXL baseline 73.3 and close to SDXL self-generated 80.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SELMA to SD v1.4 | SD v1.4 · DSG | accuracy: +4.0% | Helped |
| Apply SELMA to SD v1.4 | SD v1.4 · TIFA | accuracy: +2.9% | Helped |
| Apply SELMA to SD v2 | SD v2 · DSG | accuracy: 7.4% | Helped |
| Apply SELMA to SDXL | SDXL · DSG | accuracy: 6.9% | Helped |
| Use separate skill-specific LoRA experts and merge them rather than train a single LoRA on multiple datasets (ablation) | SD v2 · DSG | accuracy: 75.1 -> 76.9; 74.4 -> 77.7 | Helped |
| Add Localized Narratives data to a single-LoRA training mixture (ablation) | SD v2 single LoRA · DSG | accuracy: 70.3 -> 76.4 | Helped |
| Add CountBench data to a single-LoRA training mixture (ablation) | SD v2 single LoRA · DSG | accuracy: 70.3 -> 76.0 | Helped |
| Add DiffusionDB data to a single-LoRA training mixture (ablation) | SD v2 single LoRA · DSG | accuracy: 70.3 -> 73.0 | Helped |
| Add Whoops data to a single-LoRA training mixture (ablation) | SD v2 single LoRA · DSG | accuracy: 70.3 -> 73.0 | Helped |
| Add COCO data to a single-LoRA training mixture (ablation) | SD v2 single LoRA · DSG | accuracy: 70.3 -> 76.0 | Helped |
| Train a single LoRA on three datasets instead of one skill-specific dataset (ablation) | SD v2 · DSG | accuracy: 76.4 -> 75.1 | Mixed |
| Train a single LoRA on five datasets instead of one skill-specific dataset (ablation) | SD v2 · DSG | accuracy: 76.4 -> 74.4 | Mixed |
Surprising findings
- Training a single LoRA on a mixture of the auto-generated datasets degrades performance as more datasets are added, while merging separately trained experts performs best.
- DPO fine-tuning with CLIP-score-selected pairs underperformed simple LoRA merging on DSG and TIFA (75.1 vs 77.7 DSG on SD v2).
Limitations
- Relies on a strong image generator and an instruction-following LLM.
- Fine-tuning does not guarantee faithful generation for every prompt detail.
- Text rendering improvement is absent for SD v1.4 (1.8% decrease on DSG).
Experimental setup
Datasets: COCO Localized Narratives DiffusionDB CountBench Whoops DSG TIFA
Benchmarks: DSG TIFA HPS v2.1
Metrics: DSG accuracy (mPLUG VQA) TIFA accuracy (BLIP-2 VQA) PickScore ImageReward HPS human win/lose/tie
Baselines: SynGen StructureDiffusion DPOK DDPO DreamSync MoE-LoRA DPO
Builds on: LoRA ZipLoRA DreamSync DPO MoE-LoRA
Closest papers by meaning
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.87)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.87)
- Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions (CVPR 2025 · similarity 0.87)
- Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following (CVPR 2024 · similarity 0.87)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.87)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.