Research map / Image generation / Editing and style transfer
SigStyle: Signature Style Transfer via Personalized Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SigStyle, a single-image style transfer method that hypernetwork-fine-tunes the decoder attention weights of Stable Diffusion 1.4 to learn a style token and uses time-aware self-attention swapping to preserve content.
Problem
Existing style transfer methods fail to capture and transfer signature style (geometric/structural patterns, color palettes, brush strokes) from a single style image while keeping content consistent.
Approach
Treat style as an attribute learned by UNet decoder attention layers via a hypernetwork that predicts weight offsets (reducing single-image overfitting), represent the style as a learned token, and frame style transfer as reconstructing the content image with the style token, with self-attention swapping in early denoising steps for content preservation.
Key results
- Achieves the lowest Style Loss (0.7641) among compared methods and second-lowest LPIPS (0.5191) in quantitative comparison.
- User study over 1,152 votes from 32 participants gives highest Rank 1 share (9.1%) though Top 3 share (14.54%) is below CAST (17.75%).
- Tuning takes about 10 minutes and inference about 10 seconds per image on a single A6000 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SigStyle instead of the compared style-transfer methods | SigStyle · Style transfer (150 stylized images) | Style Loss: 0.7653 -> 0.7641 | Helped |
| Use SigStyle instead of AdaAttN | SigStyle · Style transfer (150 stylized images) | LPIPS: 0.4727 -> 0.5191 | Hurt |
| Use SigStyle instead of the compared style-transfer methods | SigStyle · User study (12 content-style pairs) | Rank 1 (%): 6.07 -> 9.1 | Helped |
| Use SigStyle instead of CAST | SigStyle · User study (12 content-style pairs) | Top 3 (%): 17.75 -> 14.54 | Hurt |
| Add the hypernetwork to style-aware fine-tuning (ablation) | SigStyle · Single-reference style transfer | style retention | Helped |
| Add time-aware attention swapping (ablation) | SigStyle · Style transfer content preservation | content preservation | Helped |
| Increase the attention-swapping step count k (ablation) | SigStyle · Style transfer content preservation | content similarity | Helped |
| Increase the attention-swapping step count k (ablation) | SigStyle · Style transfer style preservation | style preservation | Hurt |
| Fine-tune the decoder rather than the encoder (ablation) | Stable Diffusion UNet style inversion · Style inversion | style matching | Helped |
Surprising findings
- Fine-tuning only the UNet decoder learned style better than fine-tuning the encoder, which failed to match the reference style.
Limitations
- Requires per-style fine-tuning at inference, limiting deployment on resource-constrained devices.
- Tuning-free baselines are faster at inference (5-8 seconds).
Experimental setup
Datasets: single style reference images (e.g., The Starry Night) content images
Benchmarks: randomly selected 10 content x 15 style images (150 stylized images)
Metrics: LPIPS Style Loss user study Rank 1 % user study Top 3 % tuning and inference time
Baselines: StyleID InstantStyle Diffuse-IT InST AesPA-Net CAST AdaAttN ArtFlow
Builds on: DreamBooth E4T PhotoSwap SwapAnything InST
Closest papers by meaning
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.90)
- Semantix: An Energy-guided Sampler for Semantic Style Transfer (ICLR 2025 · similarity 0.89)
- StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfer (CVPR 2025 · similarity 0.89)
- OmniStyle: Filtering High Quality Style Transfer Data at Scale (CVPR 2025 · similarity 0.89)
- CSGO: Content-Style Composition in Text-to-Image Generation (NeurIPS 2025 · similarity 0.89)
- CLeaR: A Unified Framework for Resolving the Leakage–Degradation Dilemma in Style Transfer (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.