Research map / Image generation / Fast diffusion sampling
It's All About Your Sketch: Democratising Sketch Control in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an abstraction-aware sketch-to-image diffusion framework that replaces spatial sketch conditioning with a learned sketch adapter producing text-like embeddings, trained with FG-SBIR discriminative loss, super-concept regularisation, and abstraction-aware timestep sampling.
Problem
Existing sketch-conditioned diffusion models (ControlNet, T2I-Adapter, SGDM) directly translate sketch contours spatially, producing deformed, edge-bleeding outputs for amateur freehand sketches and requiring careful text prompts and weight tuning.
Approach
Convert the input sketch into an equivalent textual-style embedding via a sketch adapter fed into cross-attention (no spatial conditioning and no text at inference), and add a frozen fine-grained SBIR discriminative loss on Tweedie-estimated clean images plus a sketch-abstraction-dependent timestep sampling scheme.
Key results
- Proposed method achieves FID-C 16.20 and FGM 0.81 on Sketchy, vs. ControlNet's 21.22 and 0.62 FGM.
- Human MOS study gives the proposed method 4.52 vs. 3.68 for ControlNet.
- Ablation: removing the sketch adapter drops FGM from 0.81 to 0.41 (about 49% drop).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed sketch adapter instead of convolutional and FC layers (ablation) | Proposed sketch-to-image method · Sketchy | FGM: 49.38% | Helped |
| add fine-grained discriminative learning (ablation) | Proposed sketch-to-image method · Sketchy | FGM: 44.44% | Helped |
| add super-concept preservation loss (ablation) | Proposed sketch-to-image method · Sketchy | FGM: 17.28% | Helped |
| add abstraction-aware time-step sampling (ablation) | Proposed sketch-to-image method · Sketchy | FID-I: 26.64% | Helped |
| use the proposed sketch-to-image method rather than ControlNet | Sketch-to-image generation · Sketchy | FGM: 23.45% | Helped |
| use the proposed sketch-to-image method rather than ControlNet | Sketch-to-image generation · Sketchy | MOS: 1.36 ± 0.2 point improvement | Helped |
| use image captioning rather than class-label prompts | B-Captioning versus B-Classification · Sketchy | FGM: 0.36 -> 0.47 | Helped |
Surprising findings
- Passing a null text prompt substantially degrades outputs of existing sketch-conditioned diffusion models (ControlNet, T2I-Adapter, SGDM).
Limitations
- Categorical ambiguity between visually similar classes (e.g., apple vs. pear, zebra vs. horse) in abstract sketches
- Evaluated mainly on Sketchy with 125 categories; relies on a fixed class-specific prompt for baselines
- SGDM only compared qualitatively since code/weights unavailable
Experimental setup
Datasets: Sketchy TU-Berlin QuickDraw
Benchmarks: Sketchy
Metrics: FID-I FID-C FGM MOS
Baselines: ControlNet T2I-Adapter PITI SGDM pix2pix CycleGAN B-Classification B-Captioning
Builds on: Latent Diffusion Model ControlNet T2I-Adapter CLIP for All Things Zero-Shot Sketch-Based Image Retrieval
Closest papers by meaning
- Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch (CVPR 2025 · similarity 0.89)
- SketchINR: A First Look into Sketches as Implicit Neural Representations (CVPR 2024 · similarity 0.88)
- SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models (CVPR 2025 · similarity 0.88)
- How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? (CVPR 2024 · similarity 0.88)
- Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers (CVPR 2024 · similarity 0.87)
- Latent Representation Matters: Human-like Sketches in One-shot Drawing Tasks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.