Research map / Image generation / Editing and style transfer
One-Shot Reference-based Structure-Aware Image to Sketch Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ref2Sketch-SA, a training-free method that uses DDIM inversion, frequency-based noise injection, and IP-Adapter style injection on SDXL to turn a content image into a sketch matching a single reference sketch's style.
Problem
Reference-based sketch generation: rendering a content image as a sketch that matches the line and texture style of one reference sketch, while preserving the content structure, without paired training data.
Approach
Combines DDIM inversion for structural consistency with a Otsu-based high/low frequency texture-enhancement module that blends noise by region, plus IP-Adapter style features injected into style-sensitive attention layers and an acutance (Sobel) regularization during denoising, all without ControlNet or fine-tuning.
Key results
- On 4SKST, Ours achieves LPIPS 0.1058, PSNR 36.14, FID 81.57, best on all metrics versus Semi-ref2sketch (0.1271, 35.58, 82.18).
- On the Anime dataset, Ours reaches LPIPS 0.3702, PSNR 15.30, FID 165.40, SSIM 0.8217, outperforming Semi-ref (SSIM 0.7848).
- In a 270-participant user study, Ours received 59.01% of style preference votes versus 12.95% for the next best method (Semi-ref).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ref2Sketch-SA instead of Semi-ref2sketch | reference-based sketch generation · 4SKST | LPIPS: 0.1271 -> 0.1058 | Helped |
| Use Ref2Sketch-SA instead of Semi-ref2sketch | reference-based sketch generation · 4SKST | PSNR: 35.58 -> 36.14 | Helped |
| Use Ref2Sketch-SA instead of Semi-ref | reference-based sketch generation · Anime | LPIPS: 0.4623 -> 0.3702 | Helped |
| Use Ref2Sketch-SA instead of Semi-ref | reference-based sketch generation · Anime | PSNR: 13.24 -> 15.30 | Helped |
| Add DDIM inversion (ablation) | Ref2Sketch-SA · FS2K | LPIPS: 0.5104 -> 0.4018 | Helped |
| Add Texture Enhanced Module (ablation) | Ref2Sketch-SA · FS2K | LPIPS: 0.4892 -> 0.4018 | Helped |
| Add Acutance Regularization (ablation) | Ref2Sketch-SA · FS2K | LPIPS: 0.4750 -> 0.4018 | Helped |
| Add Layout Injection (ablation) | Ref2Sketch-SA · FS2K | LPIPS: 0.4328 -> 0.4018 | Helped |
Limitations
- The frequency-split parameter beta must be adjusted manually per image.
- Quantitative style evaluation lacks suitable metrics, so relies heavily on a user study.
- Standard pixel/perceptual metrics do not fully capture sparse sketch representations.
Experimental setup
Datasets: 4SKST FS2K Anime APDrawings
Benchmarks: 4SKST Anime FS2K APDrawings
Metrics: LPIPS PSNR FID SSIM user preference score
Baselines: Ref2Sketch Semi-ref2Sketch StyleID InstantStyle IP-Adapter SketchKeras IrwGAN SwappingAutoencoder MUNIT Infor-drawing APDrawingGAN UPDG
Builds on: SDXL IP-Adapter InstantStyle DDIM inversion MuGE
Closest papers by meaning
- Text to Sketch Generation with Multi-Styles (NeurIPS 2025 · similarity 0.91)
- It's All About Your Sketch: Democratising Sketch Control in Diffusion Models (CVPR 2024 · similarity 0.87)
- Image Referenced Sketch Colorization Based on Animation Creation Workflow (CVPR 2025 · similarity 0.87)
- LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model (CVPR 2025 · similarity 0.86)
- StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfer (CVPR 2025 · similarity 0.86)
- Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.