Research map / Image generation / Editing and style transfer
Text to Sketch Generation with Multi-Styles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M3S, a training-free diffusion framework for text-guided sketch generation with single or multiple reference style sketches, using linearly blended K/V injection, style-content guidance, and a joint AdaIN module for style tendency control.
Problem
Existing sketch generation methods offer limited control over sketch style, and K/V-substitution style transfer causes content leakage and artifacts when reference and target sketches are structurally dissimilar.
Approach
Concatenate reference K/V features with target K/V, linearly blend them with target features (lambda smoothing) instead of overwriting or AdaIN-aligning them, split classifier-free guidance into separate content and style directions, and mix multiple references with a joint AdaIN weight eta on latents.
Key results
- M3S (SDXL) achieves the highest average CLIP-T score of 0.3514 under default settings across styles, indicating better text alignment.
- In a human preference study (44 valid responses, ranking 1-8), M3S (SDXL) scored 6.19 and M3S (SD v1.5) 5.44, above InstantStyle at 5.08.
- Ablation: lambda=0.1 gives best CLIP-T (0.3473) and a reasonable DINO score (0.6459), while plain K/V swap yields DINO 0.3120 CLIP-T... with DINO 0.8109 but low text alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M3S (SDXL) instead of the strongest CLIP-T baseline | Sketch generation · Styles 1–6 | CLIP-T | Mixed |
| Use M3S (SDXL) instead of AttentionDistillation | Sketch generation · Styles 1–6 | DINO | Hurt |
| Use M3S (SDXL) instead of the strongest human-rating baseline | Sketch generation · Human preference assessment | rating: 6.19 vs 5.08 | Helped |
| Replace direct K/V swap with auxiliary reference features (λ=0) (ablation) | M3S · Styles 1–4 average | CLIP-T: 0.3120 -> 0.3409 | Helped |
| Replace direct K/V swap with auxiliary reference features (λ=0) (ablation) | M3S · Styles 1–4 average | DINO: 0.8109 -> 0.7437 | Hurt |
| Increase λ from 0 to 0.1 (ablation) | M3S linear feature smoothing · Styles 1–4 average | CLIP-T: 0.3409 -> 0.3473 | Helped |
| Increase λ from 0 to 0.1 (ablation) | M3S linear feature smoothing · Styles 1–4 average | DINO: 0.7437 -> 0.6459 | Hurt |
| Increase λ from 0 to 0.1 (ablation) | M3S linear feature smoothing · Styles 1–4 average | aesthetic: 4.7952 -> 4.9488 | Helped |
| Increase λ from 0.1 to 1 (ablation) | M3S linear feature smoothing · Styles 1–4 average | CLIP-T: 0.3473 -> 0.3382 | Hurt |
| Increase λ from 0.1 to 1 (ablation) | M3S linear feature smoothing · Styles 1–4 average | DINO: 0.6459 -> 0.3707 | Hurt |
| Increase λ from 0.1 to 1 (ablation) | M3S linear feature smoothing · Styles 1–4 average | aesthetic: 4.9488 -> 5.0493 | Helped |
| Use style-oriented SDXL settings (lower ω1 and higher ω2) (ablation) | M3S (SDXL) · Styles 1–5 | DINO: 0.6545 -> 0.7344 (Style1); 0.6531 -> 0.7356 (Style2); 0.6041 -> 0.6870 (Style3); 0.6493 -> 0.7212 (Style4); 0.5332 -> 0. | Helped |
Surprising findings
- Plain K/V swap achieves the highest DINO style score but the lowest CLIP-T, showing a strong style-text trade-off.
- M3S (SDXL) struggles to use abstract QuickDraw references, scoring lower DINO-ref1 than DINO-ref2 in QD-S5 pairs.
Limitations
- Fails when reference sketches are very sparse or small in the image, since AdaIN modulation and style extraction break down.
- Style-content guidance weights (omega_1, omega_2) and lambda require manual calibration per style.
- Each sketch takes about 40 s (SD v1.5) and 70 s (SDXL) on an A100 40GB; style control is global rather than localized.
Experimental setup
Datasets: 4skst Sketchy QuickDraw web-collected Style 5 set
Benchmarks: 4skst styles 1-4 web-collected Style 5 Sketchy abstract Style 6
Metrics: CLIP-T DINO similarity VGG Gram-matrix style loss aesthetic score human preference rating
Baselines: StyleAligned AttentionDistillation CSGO StyleStudio RB-Modulation InstantStyle
Builds on: StyleAligned Stable Diffusion classifier-free guidance DDIM null-text inversion
Closest papers by meaning
- One-Shot Reference-based Structure-Aware Image to Sketch Synthesis (AAAI 2025 · similarity 0.91)
- StyleStudio: Text-Driven Style Transfer with Selective Control of Style Elements (CVPR 2025 · similarity 0.88)
- SketchAgent: Language-Driven Sequential Sketch Generation (CVPR 2025 · similarity 0.88)
- Style Aligned Image Generation via Shared Attention (CVPR 2024 · similarity 0.87)
- SigStyle: Signature Style Transfer via Personalized Text-to-Image Models (AAAI 2025 · similarity 0.87)
- DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.