Research map / Image generation / Editing and style transfer
Semantix: An Energy-guided Sampler for Semantic Style Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Semantic Style Transfer and Semantix, a training-free energy-guided diffusion sampler that transfers style and appearance from a reference to image or video content via semantic correspondence.
Problem
Existing visual transfer methods handle global style transfer and local appearance transfer separately, ignoring semantic correspondence between reference and context, and video continuity suffers when image methods are applied frame by frame.
Approach
An energy function guides diffusion sampling (no training) with three terms: style feature guidance using DIFT-based semantic correspondence plus position encoding, spatial feature guidance on context features, and a cross-attention regularisation after swapping self-attention keys/values; applied to both Stable Diffusion 1.5 (images) and AnimateDiff (videos).
Key results
- On 1000 COCO-WikiArt pairs, Semantix reports LPIPS 0.461 and SSIM 0.589, outperforming the best baseline CAST (LPIPS 0.465) and best SSIM baseline TI-Guided/StyTR2 (0.561).
- Semantix achieves the lowest Gram style loss (2.524e-2) and highest PickScore (19.95) and HPS (18.78) among compared image methods.
- On video evaluation, Semantix attains Semantic Consistency 0.944 and Object Consistency 0.955, the best among MCCNet, UNIST, Cross-Image and CCPL, though its Motion Alignment (-1.894) is second to CCPL (-1.792).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Style Feature Guidance (ablation) | Semantix · COCO-WikiArt image style transfer | Gram Metrics ×10²: 4.2289 -> 2.5242 | Helped |
| add Spatial Feature Guidance (ablation) | Semantix · COCO-WikiArt image style transfer | LPIPS: 0.512 -> 0.461 | Helped |
| add Position Encoding (ablation) | Semantix · COCO-WikiArt image style transfer | Gram Metrics ×10²: 2.8982 -> 2.5242 | Helped |
| add Semantic Distance regularization (ablation) | Semantix · COCO-WikiArt image style transfer | Gram Metrics ×10²: 2.8610 -> 2.5242 | Helped |
| use 3D instead of 2D Position Encoding (ablation) | Semantix video style transfer · Video style transfer | Style similarity | Hurt |
Surprising findings
- Randomly disrupting the feature correspondences causes only a slight drop in image quality, suggesting the method does not strictly require accurate semantic correspondence.
- Removing the style feature guidance term lowers LPIPS and CFSD in the ablation (0.406 vs 0.461), which is counter to the expectation that style guidance is needed for structure.
Limitations
- Less effective when the context visual already has strong inherent style features, which can cause failures.
- Relies on semantic features from UNet-based diffusion models; not yet validated on UViT or DiT architectures.
Experimental setup
Datasets: COCO WikiArt Cross-Image appearance images Sora-generated videos
Benchmarks: COCO-WikiArt context-style pairs (1000) Cross-Image appearance domains (11 domains) 100 Sora video set
Metrics: LPIPS CFSD SSIM Gram matrix style loss PickScore HPS Semantic Consistency Object Consistency Motion Alignment Visual Quality
Baselines: AdaAttn StyleFormer StyTR2 CAST CCPL InST Cross-Image StyleID DreamStyler TI-Guided-Edit MCCNet UNIST
Builds on: DDPM inversion DIFT Cross-Image classifier-free guidance AdaIN
Closest papers by meaning
- SigStyle: Signature Style Transfer via Personalized Text-to-Image Models (AAAI 2025 · similarity 0.89)
- SCSA: A Plug-and-Play Semantic Continuous-Sparse Attention for Arbitrary Semantic Style Transfer (CVPR 2025 · similarity 0.89)
- CLeaR: A Unified Framework for Resolving the Leakage–Degradation Dilemma in Style Transfer (NeurIPS 2026 · similarity 0.88)
- OmniStyle: Filtering High Quality Style Transfer Data at Scale (CVPR 2025 · similarity 0.87)
- StyleMaster: Stylize Your Video with Artistic Generation and Translation (CVPR 2025 · similarity 0.87)
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.