Research map / Image generation / Editing and style transfer
SCSA: A Plug-and-Play Semantic Continuous-Sparse Attention for Arbitrary Semantic Style Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCSA, a training-free plug-and-play attention module combining semantic continuous attention and semantic sparse attention that makes attention-based arbitrary style transfer methods (SANet, StyTr2, StyleID) respect semantic region correspondence.
Problem
Attention-based arbitrary style transfer methods produce style inconsistencies when content and style images share the same semantics, because universal attention ignores semantic region membership and over-weights structurally similar points across regions.
Approach
Replace universal attention with two semantic-masked attentions: a continuous attention over semantic-map features that gives every query all keys within its semantic region (overall style), and a sparse attention that keeps only the single most similar key per query within the region (vivid texture), combined with semantic AdaIN preprocessing and a weighted fusion.
Key results
- SANet with SCSA reaches SSL 0.8762 vs 1.6583 for plain SANet, and the best SSL among all compared methods.
- StyTr2 with SCSA gets FID 12.3963 (vs 12.5273 plain) and CFSD 0.0705, the lowest CFSD among SOTA methods.
- In user study, SCSA-enhanced Attn-AST methods are preferred at 0.8315 (SANet), 0.8424 (StyTr2), and 0.7808 (StyleID) against their plain versions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SCSA to SANet | SANet · Semantic style transfer evaluation | SSL ↓: 1.6583 -> 0.8762 | Helped |
| Add SCSA to StyTr2 | StyTr2 · Semantic style transfer evaluation | SSL ↓: 1.9826 -> 1.2228 | Helped |
| Add SCSA to StyleID | StyleID · Semantic style transfer evaluation | SSL ↓: 1.7538 -> 1.2447 | Helped |
| Add SSA (ablation) | SANet + SCSA · Semantic style transfer evaluation | SSL ↓: 0.8840 -> 0.8762 | Helped |
| Add SCA (ablation) | SANet + SCSA · Semantic style transfer evaluation | SSL ↓: 0.9096 -> 0.8762 | Helped |
| Add S-AdaIN (ablation) | SANet + SCSA · Semantic style transfer evaluation | SSL ↓: 0.8769 -> 0.8762 | No clear effect |
| Add SSA (ablation) | StyTr2 + SCSA · Semantic style transfer evaluation | SSL ↓: 1.4157 -> 1.2228 | Helped |
| Add SCA (ablation) | StyTr2 + SCSA · Semantic style transfer evaluation | SSL ↓: 1.5714 -> 1.2228 | Helped |
| Add S-AdaIN (ablation) | StyTr2 + SCSA · Semantic style transfer evaluation | SSL ↓: 1.2832 -> 1.2228 | Helped |
| Add SSA (ablation) | StyleID + SCSA · Semantic style transfer evaluation | SSL ↓: 1.3302 -> 1.2447 | Helped |
| Add SCA (ablation) | StyleID + SCSA · Semantic style transfer evaluation | SSL ↓: 2.2981 -> 1.2447 | Helped |
| Add S-AdaIN (ablation) | StyleID + SCSA · Semantic style transfer evaluation | SSL ↓: 1.7558 -> 1.2447 | Helped |
Surprising findings
- Diffusion-based StyleID with SCSA has slightly higher CFSD (0.0916 vs 0.1178 for plain StyleID, reduced; but CFSD of StyleID+SCSA is higher than StyleID plain in the table layout), a content-preservation trade-off the authors accept.
- Removing S-AdaIN lowers CFSD for StyTr2 and StyleID, but at the cost of stylization quality.
Limitations
- Relies on user-provided or constructed semantic maps for content and style images.
- Adds runtime and memory cost over universal attention, and CFSD slightly worse for diffusion-based StyleID.
Experimental setup
Datasets: images from prior research, datasets, and the internet with constructed semantic maps
Benchmarks: internal semantic style transfer test set (quadruples of content/style images and semantic maps)
Metrics: Semantic Style Loss (SSL) FID Content Feature Structural Distance (CFSD) user preference
Baselines: SANet StyTr2 StyleID STROTSS MAST TR DIA GLStyleNet
Builds on: SANet StyTr2 StyleID AdaIN Universal Attention
Closest papers by meaning
- HSI: A Holistic Style Injector for Arbitrary Style Transfer (CVPR 2025 · similarity 0.89)
- Semantix: An Energy-guided Sampler for Semantic Style Transfer (ICLR 2025 · similarity 0.89)
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.88)
- Z*: Zero-shot Style Transfer via Attention Reweighting (CVPR 2024 · similarity 0.87)
- OmniStyle: Filtering High Quality Style Transfer Data at Scale (CVPR 2025 · similarity 0.87)
- SigStyle: Signature Style Transfer via Personalized Text-to-Image Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.