Research map / Image generation / Text-to-image diffusion
Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free masking method that zeroes out image-feature elements (chosen by clustering element-wise products of style-reference image and content-text features) in IP-Adapter to suppress content leakage in style transfer with text-to-image diffusion models.
Problem
Style-reference image conditioning in text-to-image diffusion models causes content leakage (unwanted reference content leaks into outputs) and style degradation when style strength is reduced, and prior training-free fixes (InstantStyle feature subtraction across modalities) suffer image-text misal
Approach
Rather than subtracting cross-modal content features, mask (zero out) the image-feature elements most correlated with the content text feature, identified by k-means clustering of the element-wise product of image and text features; theory argues fewer, well-chosen conditions yield lower divergence ("less is more").
Key results
- On the self-constructed dataset, the method reaches fidelity 0.972 and leakage 0.478 (lowest), versus InstantStyle at 0.820 fidelity and 0.596 leakage; InstantStyle has the top style score (0.290) vs 0.273 for the method.
- On StyleBench, the method gets 42.1% human preference, the highest among compared methods (StyleShot 21.1%, DEADiff 10.1%, InstantStyle 7.9%).
- Ablation on clustering number K shows K=2 gives the highest text alignment on 3D Model, Anime, and Baroque styles (e.g., 0.213, 0.234, 0.257).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mask content-related image-feature elements instead of subtracting content text features | Style-transfer diffusion models · Constructed style-transfer dataset | fidelity score: 0.956 -> 0.972 | Helped |
| Mask content-related image-feature elements instead of subtracting content text features | Style-transfer diffusion models · Constructed style-transfer dataset | leakage score: 0.523 -> 0.478 | Helped |
| Mask content-related image-feature elements instead of subtracting content text features | Style-transfer diffusion models · Constructed style-transfer dataset | style score: 0.290 -> 0.273 | Mixed |
| Mask content-related image-feature elements instead of subtracting content text features | Style-transfer diffusion models · StyleBench | Human Preference: 21.1% -> 42.1% | Helped |
| Set clustering number K to 2 rather than K=3, K=4, or K=5 (ablation) | Ours · StyleBench, 3D Model; Anime; Baroque | text alignment: 3D Model: 0.206/0.189/0.188 -> 0.213; Anime: 0.232/0.231/0.229 -> 0.234; Baroque: 0.253/0.253/0.252 -> 0.257 | Helped |
| Set clustering number K to 2 rather than K=3, K=4, or K=5 (ablation) | Ours · StyleBench, 3D Model; Anime; Baroque | image alignment: 3D Model: 0.478/0.485/0.487 -> 0.474; Anime: 0.381/0.390/0.380 -> 0.372; Baroque: 0.393/0.404/0.411 -> 0.384 | Mixed |
| Retain the elements identified for masking and mask the other image-feature elements (ablation) | Ours · StyleBench | style similarity | Hurt |
| Use an Image-Adapter conditioned on fewer conditions instead of a Text-Adapter (ablation) | Tuning-based style-transfer models · Constructed evaluation dataset | style score | Helped |
| Use an Image-Adapter conditioned on fewer conditions instead of a Text-Adapter (ablation) | Tuning-based style-transfer models · Constructed evaluation dataset | fidelity score | Helped |
Surprising findings
- Feature subtraction across modalities (InstantStyle) causes image-text misalignment, so reducing conditions (masking) outperforms subtraction even though subtraction looks more principled.
- The tuning-based Image-Adapter trained with fewer conditions outperforms the Text-Adapter in both style and fidelity scores, and Text-Adapter causes style degradation as training proceeds.
Limitations
- Relies on choosing the content-related elements via clustering and a fixed number of clusters (K=2), with a masking proportion that must be chosen.
- Method for unknown content relies on a generic CLIP text feature ('person, animal, plant, or object in the foreground') rather than explicit content knowledge.
- Evaluation metrics (fidelity, leakage, style scores) depend on CLIP and a limited set of 10 CIFAR-10 content classes.
Experimental setup
Datasets: CIFAR-10 StyleBench
Benchmarks: StyleBench Self-constructed CIFAR-10 style dataset
Metrics: style score fidelity score leakage score human preference text alignment image alignment
Baselines: StyleCrafter StyleAligned StyleDrop DEADiff InstantStyle StyleShot RB-Modulation CSGO InST IP-Adapter
Builds on: IP-Adapter InstantStyle classifier-free guidance energy diffusion guidance
Closest papers by meaning
- StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfer (CVPR 2025 · similarity 0.89)
- Z*: Zero-shot Style Transfer via Attention Reweighting (CVPR 2024 · similarity 0.88)
- CLeaR: A Unified Framework for Resolving the Leakage–Degradation Dilemma in Style Transfer (NeurIPS 2026 · similarity 0.88)
- SigStyle: Signature Style Transfer via Personalized Text-to-Image Models (AAAI 2025 · similarity 0.88)
- Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.88)
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.