Research map / Image generation / Text-to-image diffusion
FineStyle: Fine-grained Controllable Style Personalization for Text-to-image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FineStyle, a few-shot fine-tuning method for Muse that uses concept-oriented data scaling (cross-attention-derived concept masks) and key/value-kernel adapters to enable fine-grained style control and reduce content leakage from a single style reference image.
Problem
Few-shot or one-shot style personalization of text-to-image models leaks unwanted reference-image content into generations and gives poor fine-grained control over style attributes, with prior fixes requiring costly iterative synthetic-data fine-tuning.
Approach
Decompose a single style reference image-text pair into concept-oriented sub-pairs by masking concept regions identified via aggregated cross-attention maps, and apply a parameter-efficient adapter to the key and value kernels of cross-attention layers instead of hidden features.
Key results
- FineStyle achieves a Text CLIP score of 0.314 versus 0.297 for StyleDrop, with Style score 0.661 between Muse (0.552) and StyleDrop (0.708).
- In human evaluation, FineStyle is preferred over StyleDrop for Text alignment (67.1% vs 10.8%) and Structure/Common Sense (65.2% vs 23.6%), but StyleDrop wins on Style (43.5% vs 28.9%).
- Ablation shows the kv-kernel adapter alone yields a higher Text score (0.308) than the feature adapter with data scaling (0.296), indicating kv adapters better disentangle fine-grained styles.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FineStyle instead of StyleDrop | Muse style personalization · Parti prompts, 10 styles | Text CLIP score: 0.297 -> 0.314 | Helped |
| use FineStyle instead of StyleDrop | Muse style personalization · Human evaluation | Text preference: 10.8% -> 67.1% | Helped |
| use FineStyle instead of StyleDrop | Muse style personalization · Human evaluation | Style preference: 43.5% -> 28.9% | Hurt |
| use FineStyle instead of StyleDrop | Muse style personalization · Human evaluation | Structure / Common Sense preference: 23.6% -> 65.2% | Helped |
| remove concept-oriented data scaling (ablation) | FineStyle · Parti prompts, 10 styles | Text CLIP score: 0.308 -> 0.314 | Hurt |
| replace the key/value kernel adapter with a feature adapter (ablation) | FineStyle · Parti prompts, 10 styles | Text CLIP score: 0.296 -> 0.314 | Hurt |
| replace the key/value kernel adapter with a feature adapter (ablation) | FineStyle · Parti prompts, 10 styles | Style CLIP score: 0.661 -> 0.730 | Mixed |
| increase style guidance λ1 (ablation) | FineStyle inference · cliff bay generation prompt without boats | content leakage | Hurt |
| increase semantic guidance λ2 (ablation) | FineStyle inference · cliff bay generation prompt without boats | prompt alignment | Helped |
Surprising findings
- Higher CLIP style scores can signal overfitting or content leakage rather than better style fidelity, so the Style metric is non-monotonic in quality.
- Concept-oriented data scaling with the feature adapter lowered the Text score, and kv-only adapter without data scaling performed better on Text than the combined variant with feature adapter.
Limitations
- Does not claim to handle every style composition defined by a text prompt; limited to the evaluated fine-grained properties (color, texture, foreground objects, background).
- No error bars or statistical significance reported due to compute quota.
- Concept decomposition requires a comprehensive caption, so quality depends on user or vLLM-provided concept phrases.
Experimental setup
Datasets: StyleDrop evaluation style set (24 styles) Parti prompts (filtered, 190 prompts)
Benchmarks: Parti prompts (filtered) with StyleDrop style set
Metrics: CLIP Text score CLIP Style score human preference (Style, Text, Structure/Common Sense)
Baselines: StyleDrop StyleAligned IP-Adapter DreamStyler Muse
Builds on: Muse StyleDrop MaskGIT LoRA
Closest papers by meaning
- StyleStudio: Text-Driven Style Transfer with Selective Control of Style Elements (CVPR 2025 · similarity 0.88)
- Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation (CVPR 2025 · similarity 0.88)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.88)
- Style Aligned Image Generation via Shared Attention (CVPR 2024 · similarity 0.87)
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.87)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.