Research map / Image generation / Editing and style transfer
ArtAdapter: Text-to-Image Style Transfer using Multi-Level Style Encoder and Explicit Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ArtAdapter, a zero-shot and fast-finetunable text-to-image style transfer framework on Stable Diffusion 1.5 that uses a multi-level VGG style encoder, explicit adaptation of style-only K/V projections, and a training-only auxiliary content adapter.
Problem
Text-to-image style transfer struggles to capture high-level style (composition, artistic expression) while keeping text fidelity, and per-style finetuning methods overfit to style-reference content, especially from a single reference.
Approach
Extracts multi-level VGG mean/std style features (relu3_3, relu4_3, relu5_3) into learnable style tokens, adapts only the style-side key/value projections in cross-attention via a LoRA-like residual while freezing the text pathway, uses a training-only T2I adapter with color augmentation to supply weak content structure so style features do not copy content, and adds a rank-1 residual vector for ~
Key results
- Single-reference: style similarity 0.707 and aesthetics 5.601 with fine-tuning (text 0.255), versus T2I-Style-Adapter's 0.822 style score at 0.181 text similarity.
- Multi-reference user study (212 participants): text 4.74, style 4.76, quality 4.43 for ArtAdapter vs. TI (4.49/4.35/4.38) and LoRA (3.91/4.08/4.18).
- Multi-reference CLIP metrics are roughly on par with TI and LoRA (style 0.680 vs 0.678 and 0.678).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add fast finetuning to ArtAdapter | ArtAdapter, single-reference style transfer · Single reference | Style similarity: 0.656 -> 0.707 | Helped |
| Add fast finetuning to ArtAdapter | ArtAdapter, single-reference style transfer · Single reference | Text similarity: 0.269 -> 0.255 | Hurt |
| Add fast finetuning to ArtAdapter | ArtAdapter, single-reference style transfer · Single reference | Aesthetics score: 5.532 -> 5.601 | Helped |
| Add the Auxiliary Content Adapter during training (ablation) | ArtAdapter · Style transfer | Content separation | Helped |
| Add Explicit Adaptation in the cross-attention layers (ablation) | ArtAdapter · Style transfer | Style fidelity | Helped |
| Adapt the diffusion backbone in addition to the multi-level style encoder (ablation) | ArtAdapter · Style transfer | Style representation | Helped |
Surprising findings
- T2I-Style-Adapter achieves higher style and aesthetic CLIP scores than ArtAdapter but with much lower text similarity, showing that CLIP style scores alone can reward content-ignoring outputs.
Limitations
- Style mixing is imperfect: high-level style embeddings often inadvertently carry lower-level elements, causing interference.
- Single-reference zero-shot text similarity trades off against style similarity; ArtAdapter is below some baselines on style/aesthetic CLIP scores.
- Evaluation relies on a small in-house test set of 35 prompts and a 212-person user study.
Experimental setup
Datasets: LAION Aesthetics WikiArt
Benchmarks: in-house test set of 35 prompts x 10 styles (700 pairs)
Metrics: CLIP text similarity CLIP style similarity aesthetics score user study 7-point scale
Baselines: CAST StyTr2 T2I-Style-Adapter InST FreeDoM Textual Inversion LoRA DALL-E 3
Builds on: Stable Diffusion LoRA T2I-Adapter Textual Inversion VGG
Closest papers by meaning
- OmniStyle: Filtering High Quality Style Transfer Data at Scale (CVPR 2025 · similarity 0.88)
- StyleStudio: Text-Driven Style Transfer with Selective Control of Style Elements (CVPR 2025 · similarity 0.87)
- ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank (AAAI 2024 · similarity 0.87)
- HSI: A Holistic Style Injector for Arbitrary Style Transfer (CVPR 2025 · similarity 0.87)
- SigStyle: Signature Style Transfer via Personalized Text-to-Image Models (AAAI 2025 · similarity 0.86)
- Style Aligned Image Generation via Shared Attention (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.