Research map / Image generation / Editing and style transfer
Learning to Manipulate Artistic Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIM-Net, a zero-shot exemplar-based artistic image manipulation network that uses semantic-free masks, keypoint-based warp fields, and a self-supervised region transportation strategy with pseudo ground truth.
Problem
Exemplar-based image translation for artistic images fails because accurate semantics are hard to extract from artwork, feature compression gives imprecise local control, high-resolution generation is computationally costly, and training-data priors cause cross-domain artifacts.
Approach
Replace semantic inputs with semantic-free SAM masks, align masks via a few keypoints with local affine transformations dilated into full-resolution warp fields, and merge warped exemplar regions (region transportation) with a texture-guided pseudo ground truth and style self-supervision, avoiding generative priors from realistic training data.
Key results
- SIM-Net achieves overall Style Loss 1.31 (ROI 1.62) versus 2.01 for the best prior method MCL-Net, with SSIM 0.95 and LPIPS 0.031.
- SIM-Net w DM reaches PSNR 30.124 and LPIPS 0.028 overall, compared with 25.688 PSNR and 0.087 LPIPS for MCL-Net.
- Inference time of 0.126 s versus 0.301 s for MATEBIT and 0.574 s for MCL-Net on 20 high-resolution images, with memory 89.64 versus 59.3 for CoCosNet v2 (lowest).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add L_bound (ablation) | SIM-Net · Ablation study | SSIM: 0.79 -> 0.97 | Helped |
| Add L_context (ablation) | SIM-Net · Ablation study | Style Loss: 2.24 -> 1.34 | Helped |
| Add L_cyc (ablation) | SIM-Net · Ablation study | SSIM: 0.72 -> 0.97 | Helped |
| Replace semantic mask with semantic-free mask (ablation) | SIM-Net · Ablation study | Style Loss: 1.09 -> 1.34 | Hurt |
| Replace semantic mask with semantic-free mask (ablation) | SIM-Net · Ablation study | SSIM: 0.96 -> 0.97 | Helped |
| Add Style Self-Supervision (ablation) | SIM-Net · Ablation study | Style Loss: 10.16 -> 1.34 | Helped |
| Replace generation model with the proposed generation network (ablation) | SIM-Net · Ablation study | Style Loss: 8.19 -> 1.34 | Helped |
| Add GAN loss (ablation) | SIM-Net · Artistic image manipulation | local manipulation sensitivity | Hurt |
Surprising findings
- Adding a GAN loss made results insensitive to local manipulation, so the authors dropped adversarial training.
- Replacing the semantic mask with a semantic-free mask yields a lower Style Loss (1.09) than the full model (1.34) in the ablation table.
Limitations
- Less creative than generative models; suited to precisely controlled scenarios.
- Affects artistic aesthetics such as brush strokes, an open challenge.
- Splicing artifacts from region transportation require the texture-guidance module to remove.
Experimental setup
Datasets: CelebAMask-HQ Weizmann Horse Database Intel Image Classification
Benchmarks: 237 artistic test images across 10 styles
Metrics: Style Loss SSIM PSNR LPIPS color similarity texture similarity fake detection (MVSS-Net++) inference time memory
Baselines: INADE Swapping autoencoder (PZ20) PMD LY21 CoCosNet UNITE CoCosNet v2 MCL-Net DynaST MATEBIT
Builds on: CoCosNet Segment Anything BoundaryIoU loss Unsupervised keypoint-based motion transfer
Closest papers by meaning
- ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation (NeurIPS 2023 · similarity 0.88)
- Paint by Example: Exemplar-Based Image Editing With Diffusion Models (CVPR 2023 · similarity 0.88)
- Cross-Domain Correspondence Learning for Exemplar-Based Image Translation (CVPR 2020 · similarity 0.87)
- SinIR: Efficient General Image Manipulation with Single Image Reconstruction (ICML 2021 · similarity 0.87)
- Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation (CVPR 2025 · similarity 0.87)
- Zero-shot Image Editing with Reference Imitation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.