Research map / Image generation / Editing and style transfer
Training-and-Prompt-Free General Painterly Harmonization via Zero-Shot Disentenglement on Style and Content References
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TF-GPH, a training-free, prompt-free diffusion pipeline for general painterly harmonization (object insertion, swapping, style transfer) using a similarity disentangle mask and similarity reweighting in shared attention, plus a new GPH Benchmark with range-based metrics.
Problem
Painterly image harmonization methods either need training on limited data or rely on text prompts, and existing attention-based training-free editing causes content disruption; evaluation also lacks diversity and fails to capture the stylization-content trade-off.
Approach
Extends shared self-attention to three images (foreground, background style reference, composite) with a similarity disentangle mask that routes composite queries to the matching reference, and scales pre-softmax similarities (alpha, beta) so that high-similarity style features are amplified, balancing stylization and content preservation without training or prompts.
Key results
- On the GPH Benchmark (512x512), the method with content-emphasis setting reaches LP_fg 0.10 and CP_img 95.42, versus ArtoPIH's LP_fg 0.37 and CP_img 84.96 for painterly harmonization.
- Similarity reweighting raises CP_dir from 0.18 (SDM only) to 11.69 and CP_st from 47.50 to 56.37 for painterly harmonization in the ablation.
- On the TF-ICON benchmark, the method reports LP_bg 0.05, LP_fg 0.48, CP_img 83.34 and CP_text 30.33, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add similarity disentangle mask (ablation) | TF-GPH · GPH Benchmark, painterly harmonization | CP_dir: 0.11 -> 0.18 | Helped |
| Add similarity disentangle mask (ablation) | TF-GPH · GPH Benchmark, style transfer | CP_st: 57.97 -> 59.05 | Helped |
| Add similarity reweighting to the similarity disentangle mask (ablation) | TF-GPH · GPH Benchmark, painterly harmonization | CP_st: 47.50 -> /56.37 | Helped |
| Add similarity reweighting to the similarity disentangle mask (ablation) | TF-GPH · GPH Benchmark, style transfer | CP_st: 59.05 -> /78.60 | Helped |
| Use a finite γ value rather than γ=-∞ (ablation) | TF-GPH with (α, β)=(0.9, 1.1) · GPH Benchmark, painterly harmonization | CP_style: 56.37 -> 52.33; 56.37 -> 48.71; 56.37 -> 47.61 | Hurt |
| Increase β from 1.1 to 1.5 while holding α=0.9 (ablation) | TF-GPH · GPH Benchmark, painterly harmonization | CP_style: 56.37 -> 55.96 | No clear effect |
| Vary α, β to (0.5,1.5) from (0.9,1.1) (ablation) | TF-GPH · GPH Benchmark, painterly harmonization | CP_style: 56.37 -> 56.48 | No clear effect |
| Vary α, β to (1.5, 1.5) from (0.9, 1.1) (ablation) | TF-GPH · GPH Benchmark, painterly harmonization | CP_style: 56.37 -> 45.64 | Hurt |
| Vary α, β to (0.5, 0.5) from (0.9, 1.1) (ablation) | TF-GPH · GPH Benchmark, painterly harmonization | CP_style: 56.37 -> 49.51 | Hurt |
| Use γ=-∞ to block composite-image attention (ablation) | TF-GPH similarity disentangle mask · GPH Benchmark, painterly harmonization | CP_style: 47.50 -> 56.37 | Helped |
| Use TF-GPH instead of TF-ICON | TF-ICON Benchmark · TF-ICON Benchmark | LP_fg: 0.60 -> 0.48 | Helped |
| Use TF-GPH instead of TF-ICON | TF-ICON Benchmark · TF-ICON Benchmark | CP_img: 82.86 -> 83.34 | Helped |
Surprising findings
- Reconstruction of the composite with the similarity disentangle mask alone yields nearly no reconstruction loss, showing the composite can be fully controlled by the two references.
Limitations
- Relies on user-chosen inference-time hyperparameters (T_share, L_share, alpha, beta) with no universally optimal setting.
- Benchmark uses only 635 human-made test cases and the user study involves 20 participants per task.
- Stylization upper bound yields large LP_fg and LP_bg in some settings, showing content-stylization trade-off.
Experimental setup
Datasets: WikiArt COCO TF-ICON Benchmark
Benchmarks: GPH Benchmark WikiArt with COCO TF-ICON Benchmark
Metrics: LPIPS (LP_bg, LP_fg) CLIP image similarity (CP_img) CLIP style similarity (CP_st) CP_dir range-based bounds user preference
Baselines: ArtoPIH ProPIH PHDiffusion TF-ICON StyleID Z-STAR StyTr2 QuantArt VCT
Builds on: Stable Diffusion DPM-Solver++ shared attention (StyleAligned) TF-ICON Z-STAR
Closest papers by meaning
- Progressive Painterly Image Harmonization from Low-Level Styles to High-Level Styles (AAAI 2024 · similarity 0.90)
- Painterly Image Harmonization by Learning from Painterly Objects (AAAI 2024 · similarity 0.88)
- Relightful Harmonization: Lighting-aware Portrait Background Replacement (CVPR 2024 · similarity 0.86)
- FRIH: Fine-Grained Region-Aware Image Harmonization (AAAI 2024 · similarity 0.86)
- Free-Lunch Color-Texture Disentanglement for Stylized Image Generation (NeurIPS 2025 · similarity 0.85)
- Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.