Research map / Image generation / Text-to-image diffusion
Stylus: Automatic Adapter Selection for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Stylus refines adapter descriptions with a VLM, retrieves top-150 LoRAs via embeddings, and uses an LLM composer to assign adapters to prompt keywords, then masks and composes them into Stable Diffusion 1.5, releasing the 75K-LoRA StylusDocs dataset.
Problem
Automatically selecting and composing relevant LoRA-style adapters for a text-to-image prompt from a huge, poorly documented adapter database (100K+ on Civitai), where prompts imply multiple tasks and composing adapters can degrade images.
Approach
A three-stage Refine-Retrieve-Compose pipeline: a VLM rewrites adapter descriptions from sample images, embeddings retrieve candidates against the full prompt, and a long-context LLM segments the prompt into keyword tasks and assigns adapters per task, followed by random binary masking and weight normalization across tasks.
Key results
- On COCO with Realistic-Vision-v6 (CFG=6), Stylus obtains CLIP 27.25 and FID 22.05, versus SD v1.5 at CLIP 27.22 and FID 23.96, and beats the reranker and retriever-only baselines on FID.
- Human raters preferred Stylus outputs about 68% of the time on realistic checkpoint comparisons, and VLM judging gave a 60% win rate on visual fidelity excluding ties.
- For diversity, Stylus wins about 60% (dFID) and 58% (GPT-4V) of prompts excluding ties across 200 prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Stylus adapter selection and composition instead of the SD v1.5 baseline | Stable Diffusion image generation · COCO 2014 validation | CLIP: 27.22 -> 27.25 (+0.03) | Helped |
| use Stylus adapter selection and composition instead of the SD v1.5 baseline | Stable Diffusion image generation · COCO 2014 validation | FID: 23.96 -> 22.05 (-1.91) | Helped |
| remove the composer stage (ablation) | Stylus · COCO 2014 validation | CLIP: 27.25 -> 24.93 | Hurt |
| remove the composer stage (ablation) | Stylus · COCO 2014 validation | FID: 22.05 -> 24.68 | Hurt |
| replace the composer with a cross-encoder reranker (ablation) | Stylus · COCO 2014 validation | CLIP: 27.25 -> 25.48 | Hurt |
| replace the composer with a cross-encoder reranker (ablation) | Stylus · COCO 2014 validation | FID: 22.05 -> 22.81 | Hurt |
| select adapters randomly instead of using Stylus retrieval and composition (ablation) | Stable Diffusion image generation · COCO 2014 validation | CLIP: 27.25 -> 26.34 | Hurt |
| select adapters randomly instead of using Stylus retrieval and composition (ablation) | Stable Diffusion image generation · COCO 2014 validation | FID: 22.05 -> 24.39 | Hurt |
| use Stylus instead of the Stable Diffusion realistic checkpoint | Stable Diffusion image generation · PartiPrompts | visual fidelity preference: 60% | Helped |
| use Stylus instead of the Stable Diffusion checkpoint | Stable Diffusion image generation · PartiPrompts | textual alignment | No clear effect |
| use Stylus instead of SD v1.5 | Stable Diffusion image generation · PartiPrompts, 200 prompts | dFID win rate: approximately 60% | Helped |
| use Stylus instead of SD v1.5 | Stable Diffusion image generation · PartiPrompts, 200 prompts | GPT-4V diversity win rate: 58% | Helped |
Surprising findings
- Retriever-only and reranker adapter selection underperform the base Stable Diffusion model, as similar-but-irrelevant adapters introduce biases.
- Random adapter selection performs close to but slightly worse than the base model, since orthogonal adapters rarely alter the prompt's concepts.
Limitations
- Most of the adapter database is character/celebrity LoRAs (about 70%), so the evaluation covers a minor share of adapters relevant to COCO prompts.
- Failure modes include task blocking, reduced within-task diversity, low-quality adapters, and retrieval/classification errors compounding across stages.
- Composer adds 12.1 s latency at batch size 1 (about 75% overhead) and relies on long-context LLM calls with 20K+ tokens.
Experimental setup
Datasets: StylusDocs (75K LoRAs from Civitai and HuggingFace) MS-COCO 2014 validation PartiPrompts
Benchmarks: MS-COCO 2014 validation (CLIP, FID) PartiPrompts (human and VLM preference)
Metrics: CLIP score FID human preference win rate VLM win rate for visual fidelity and textual alignment dFID (per-prompt diversity) GPT-4V diversity score
Baselines: Stable Diffusion v1.5 Random adapter selection Retriever-only (RAG) Cohere rerank-english-v2.0 reranker
Builds on: retrieval-augmented generation LoRA Stable Diffusion chain-of-thought prompting
Closest papers by meaning
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.87)
- X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model (CVPR 2024 · similarity 0.87)
- Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis (NeurIPS 2024 · similarity 0.86)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.86)
- FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models (CVPR 2024 · similarity 0.85)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.