Research map / Vision-language and multimodal / Human and embodied modeling
SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SketchFusion, a frozen-backbone hybrid that injects CLIP visual features into Stable Diffusion's UNet denoising path and learns a dynamic multi-layer aggregator to produce universal sketch features for retrieval, recognition, segmentation and correspondence.
Problem
Foundation model features (Stable Diffusion) are poor for abstract, sparse freehand sketches and are biased toward high-frequency components, which hurts sketch understanding especially for dense prediction tasks.
Approach
Complement SD's spatially-aware but semantically weak, high-frequency-biased features with CLIP visual features by adding CLIP features via 1D convolutions into all UNet upsampling layers during denoising, then adaptively aggregating multi-level SD features with learnable branch weights, training only these lightweight modules while keeping SD and CLIP frozen.
Key results
- Category-level ZS-SBIR on Sketchy reaches mAP@200 of 0.761 vs 0.746 for SD-PL, the prior best.
- Sketch-photo correspondence on PSC6K reaches PCK@5 of 70.31%, versus 58.00% for the self-supervised SoTA.
- Sketch-based segmentation reaches mIoU of 60.12 vs 46.45 for Sketch-a-Segmenter; sketch recognition on Quick, Draw! reaches Acc.@1 87.02% vs 86.10% for SketchXAI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the aggregation network to combine multi-scale UNet features (ablation) | SketchFusion · Sketch-photo correspondence, sketch-based segmentation, ZS-SBIR, and sketch recognition | PCK@5, mIoU, mAP@200, and Acc.@1: 54.65 vs 70.31 PCK@5; 47.52 vs 60.12 mIoU; 0.587 vs 0.761 mAP@200; 66.71 vs 87.02 Acc.@1 | Helped |
| Add learnable feature-branch weights (ablation) | SketchFusion · Sketch-photo correspondence, sketch-based segmentation, ZS-SBIR, and sketch recognition | PCK@5, mIoU, mAP@200, and Acc.@1: 68.75 vs 70.31 PCK@5; 58.13 vs 60.12 mIoU; 0.726 vs 0.761 mAP@200; 83.79 vs 87.02 Acc.@1 | Helped |
| Use learnable 1D convolutions rather than linear interpolation for CLIP feature injection (ablation) | SketchFusion · Sketch-photo correspondence, sketch-based segmentation, ZS-SBIR, and sketch recognition | PCK@5, mIoU, mAP@200, and Acc.@1: 55.97 vs 70.31 PCK@5; 47.24 vs 60.12 mIoU; 0.602 vs 0.761 mAP@200; 71.75 vs 87.02 Acc.@1 | Helped |
| Use Stable Diffusion v2.1 instead of v1.4 or v1.5 (ablation) | SketchFusion · Sketch-photo correspondence, sketch-based segmentation, ZS-SBIR, and sketch recognition | PCK@5, mIoU, mAP@200, and Acc.@1: 68.91, 69.24 -> 70.31 PCK@5; 57.63, 58.91 -> 60.12 mIoU; 0.728, 0.735 -> 0.761 mAP@200; 78.82, 79.59 -> 87.02 Acc.@1 | Helped |
| Use CLIP ViT-L/14 instead of ViT-B/16 or ViT-B/32 (ablation) | SketchFusion · Sketch-photo correspondence, sketch-based segmentation, ZS-SBIR, and sketch recognition | PCK@5, mIoU, mAP@200, and Acc.@1: 67.75, 69.41 -> 70.31 PCK@5; 56.38, 58.47 -> 60.12 mIoU; 0.743, 0.758 -> 0.761 mAP@200; 77.37, 78.84 -> 87.02 Acc.@1 | Helped |
| Fuse SD and CLIP with feature injection and aggregation rather than simple feature concatenation | SketchFusion · Sketch-based discriminative and dense prediction tasks | task metrics | Helped |
Surprising findings
- Naive SD+CLIP feature concatenation (B-SD+CLIP) lags the full method by about 35.78% on average, indicating injection and aggregation matter more than simply combining models.
- End-to-end finetuning of SD and CLIP performs drastically worse than frozen features across tasks, suggesting it destroys pretrained knowledge.
Limitations
- Segmentation dataset is a manually annotated subset of 10 Sketchy classes (about 5K tuples), limiting generality.
- Relies on frozen SD and CLIP with null text prompts, and the gains over naive combinations depend on the injection and aggregation modules.
Experimental setup
Datasets: Sketchy Sketchy-extended TU-Berlin Quick, Draw! PSC6K
Benchmarks: Sketchy TU-Berlin Quick, Draw! PSC6K
Metrics: mAP@200 P@200 Acc.@1 Acc.@5 PCK@5 PCK@10 mIoU pAcc.
Baselines: SD-PL LVM Gen-VAE SAKE CCGAN GRL SketchMate SketchGNN SketchXAI Self-Supervised (Lu et al.) WarpC WeakAlign
Builds on: Stable Diffusion CLIP Emergent correspondence from image diffusion SD-PL (Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers) triplet loss SBIR
Closest papers by meaning
- Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers (CVPR 2024 · similarity 0.89)
- How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? (CVPR 2024 · similarity 0.88)
- It's All About Your Sketch: Democratising Sketch Control in Diffusion Models (CVPR 2024 · similarity 0.88)
- Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch (CVPR 2025 · similarity 0.88)
- Open Vocabulary Semantic Scene Sketch Understanding (CVPR 2024 · similarity 0.88)
- SketchINR: A First Look into Sketches as Implicit Neural Representations (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.