Research map / Image generation / Text-to-image diffusion
Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LyCORIS, an open-source library of LoRA-family fine-tuning methods (LoCon, LoHa, LoKr, etc.) for Stable Diffusion, plus a systematic evaluation framework over 234 checkpoints with fidelity, controllability, and diversity metrics.
Problem
Fine-tuning Stable Diffusion involves many methods and hyperparameters, and there is no systematic way to integrate new methods or evaluate them across prompt types and concept categories.
Approach
Provide a unified open-source library implementing LoRA extensions (LoCon, LoHa using Hadamard products, LoKr using Kronecker products) and a multi-criteria evaluation framework that separates prompt types (training, trigger, generalization) from automatic image metrics, analyzed with SHAP over many checkpoints.
Key results
- Evaluation over 234 checkpoints (26 configurations x 3 seeds x 3 checkpoints) with 14,900 generated images per checkpoint, analyzed via SHAP with CatBoost.
- Tentative ranking: LoKr best on controllability and LoHa best on diversity in the reported table, with native fine-tuning at lr 5e-6 best on fidelity; results vary across concept categories.
- Finds increasing epochs generally improves fidelity while reducing text-image alignment, diversity, and base model preservation, with an overfit-then-generalize exception.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase training epochs (ablation) | LyCORIS fine-tuning · General concept categories | Concept fidelity | Helped |
| Increase training epochs (ablation) | LyCORIS fine-tuning · General concept categories | Text-image alignment | Hurt |
| Increase training epochs (ablation) | LyCORIS fine-tuning · General concept categories | Diversity | Hurt |
| Increase training epochs (ablation) | LyCORIS fine-tuning · General concept categories | Base model preservation | Hurt |
| Increase training epochs (ablation) | LyCORIS fine-tuning · Scenes and stuffed toys | Image similarity | Hurt |
| Increase learning rate within the tested range (ablation) | LyCORIS fine-tuning · Fine-tuning experiments | Concept fidelity | Mixed |
| Restrict trained layers to attention-only (ablation) | LyCORIS fine-tuning · Fine-tuning experiments | Image similarity | Hurt |
| Restrict trained layers to attention-only (ablation) | LyCORIS fine-tuning · Fine-tuning experiments | Other evaluation metrics | Helped |
| Add convolutional layers to the trained layers (ablation) | LyCORIS fine-tuning · Fine-tuning experiments | Evaluation metrics | No clear effect |
| Increase model capacity by increasing dimension or decreasing factor (ablation) | LoRA, LoHa, or LoKr · Fine-tuning experiments | Concept fidelity | Mixed |
| Use LoHa instead of other methods | Stable Diffusion 1.5 fine-tuning · Simple multi-concept tasks | Overall performance | Helped |
| Use LoKr with full dimension | Stable Diffusion 1.5 fine-tuning · Complex single-concept tasks | Overall performance | Helped |
Surprising findings
- Restricting fine-tuning to attention layers substantially reduces image similarity, while the effect of fine-tuning convolutional layers was hard to discern.
- Existing image quality assessment models were found unsuitable, so no quality metric was included in the primary experiments.
Limitations
- Image quality is not evaluated because available quality assessment models were judged unsuitable.
- No human evaluation was conducted due to scale costs.
- Multi-concept generation was not explored; metrics are relatively rudimentary.
Experimental setup
Datasets: custom 1,706-image concept dataset (anime characters, movie characters, scenes, stuffed toys, styles) LAION-5B (pretraining of Stable Diffusion, referenced)
Benchmarks: custom concept dataset with 14,900 generated images per checkpoint
Metrics: average cosine similarity (DINOv2) squared centroid distance (DINOv2) CLIP cosine similarity Vendi score style loss normalized rank score with SHAP
Baselines: LoRA LoHa LoKr Native fine-tuning
Builds on: LoRA FedPara KronA DreamBooth Textual Inversion
Closest papers by meaning
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.91)
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.90)
- LoRACLR: Contrastive Adaptation for Customization of Diffusion Models (CVPR 2025 · similarity 0.89)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.89)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.