Research map / Image generation / Editing and style transfer
ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ArtBank, which freezes Stable Diffusion 1.4 and trains an Implicit Style Prompt Bank of learnable matrices (projected by a Spatial-Statistical Self-Attention Module) to produce realistic artistic style transfer that preserves content structure.
Problem
Artistic style transfer methods either preserve content structure but yield artifacts and unrealistic results (small CNN/GAN models), or produce realistic images but fail to keep the content structure (large pre-trained diffusion models).
Approach
Keep the pre-trained Stable Diffusion frozen and learn per-artist-collection trainable parameter matrices (an implicit style prompt bank) that replace a placeholder token embedding as a visual prompt, with a SSAM module that uses spatial (row/column) and statistical (mean/std) attention to accelerate training, plus stochastic inversion of content-image noise at inference.
Key results
- ArtBank achieves the highest CLIP score on all seven artist collections among compared methods (e.g., 0.7321 on Van Gogh vs. 0.7244 for InST and 0.5440 for SD).
- In user preference voting (2,000 votes, 200 participants), ArtBank scored 0.679 on average preference against the other methods.
- SSAM reaches comparable optimization quality in about 400 iterations versus 1,400 iterations for SANet attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ArtBank instead of InST | ArtBank · Van Gogh, Morisot, Ukiyoe, Monet, Cezanne, Gauguin, Peploe | CLIP score: Van Gogh: 0.7244 -> 0.7321 | Helped |
| add the text prompt (ablation) | ArtBank · Van Gogh, Morisot, Ukiyoe, Monet, Cezanne, Gauguin, Peploe | CLIP score: Van Gogh: 0.7248 -> 0.7321 | Helped |
| use SSAM instead of SANet (ablation) | ArtBank · Van Gogh, Morisot, Ukiyoe, Monet, Cezanne, Gauguin, Peploe | CLIP score: Van Gogh: 0.7267 -> 0.7321 | Helped |
| use SSAM instead of AdaAttN (ablation) | ArtBank · Van Gogh, Morisot, Ukiyoe, Monet, Cezanne, Gauguin, Peploe | CLIP score: Van Gogh: 0.7310 -> 0.7321 | Helped |
| use SSAM rather than SANet during optimization (ablation) | ArtBank · style optimization | iterations to incomplete convergence: 1400 iters -> 400 iters | Helped |
Limitations
- Inference is slower than small feed-forward style transfer methods
- Requires separate training of the ISPB and SSAM for each artist collection (about two days each)
- Relies on a single backbone version (SD 1.4) without evaluation of other backbones
Experimental setup
Datasets: WikiArt DIV2K
Benchmarks: WikiArt-based artist collections (Van Gogh, Morisot, Ukiyoe, Monet, Cezanne, Gauguin, Peploe)
Metrics: CLIP score user preference score run time (sec)
Baselines: InST DiffuseIT Stable Diffusion AST CycleGAN LSeSim CUT SANet AdaAttN
Builds on: Stable Diffusion CLIP InST SANet
Closest papers by meaning
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.91)
- Z*: Zero-shot Style Transfer via Attention Reweighting (CVPR 2024 · similarity 0.90)
- SigStyle: Signature Style Transfer via Personalized Text-to-Image Models (AAAI 2025 · similarity 0.89)
- OmniStyle: Filtering High Quality Style Transfer Data at Scale (CVPR 2025 · similarity 0.88)
- Scalable Motion Style Transfer with Constrained Diffusion Generation (AAAI 2024 · similarity 0.87)
- ArtAdapter: Text-to-Image Style Transfer using Multi-Level Style Encoder and Explicit Adaptation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.