Research map / Adversarial robustness and security / Attacks on text and graphs
What Sketch Explainability Really Means for Downstream Tasks?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free, plug-in sketch explainability method that attributes model predictions to individual strokes (SLA) and stroke points (P-SLA) via differentiable rasterisation, and applies it to SBIR, assisted drawing, sketch-to-photo generation, and adversarial attacks.
Problem
Pixel-based explainability methods are ill-suited to sparse, human-drawn sketches, which are built from strokes; rasterisation of vector sketches is non-differentiable, blocking stroke-level attribution for downstream sketch models.
Approach
Make rasterisation partially differentiable at the stroke level by summing per-stroke raster images, and fully differentiable at the point level via a distance transform passed through a sigmoid, so gradients from a pre-trained model can attribute importance to strokes or vector points without retraining; use these attributions for task-driven applications.
Key results
- On SBIR, sketches with high stroke-order correlation (Corr>=0.5) reach 1.7/3.9 higher Acc.@1/Acc.@5 than low-correlation ones (Corr<=0.1) on Sketchy fine-grained retrieval.
- On QMUL-Shoe-V2 noisy stroke filtering, SLA improves Acc.@1 from 33.4 to 36.1 and P-SLA to 36.5, but trails Bhunia et al. (43.7).
- Stroke-removal adversarial attack at eps=5 on QuickDraw drops accuracy from 67.2 to 65.7 (SLA) and 65.1 (P-SLA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use high-correlation sketches (Corr ≥0.5) rather than low-correlation sketches (Corr ≤0.1) | Pre-trained CLIP prompt-learning SBIR model · Sketchy | Acc.@1: 17.6 vs. 12.7 | Helped |
| Use high-correlation sketches (Corr ≥0.5) rather than low-correlation sketches (Corr ≤0.1) | Pre-trained CLIP prompt-learning SBIR model · TU-Berlin | mAP: 55.3 -> 51.7 (SLA); 57.6 -> 50.1 (P-SLA) | Helped |
| Filter noisy strokes with SLA | Pre-trained SBIR model · QMUL-Shoe-V2 | Acc.@1: 33.4 -> 36.1 | Helped |
| Filter noisy strokes with P-SLA | Pre-trained SBIR model · QMUL-Shoe-V2 | Acc.@1: 33.4 -> 36.5 | Helped |
| Filter noisy strokes with SLA or P-SLA | Pre-trained SBIR model · QMUL-Chair-V2 | Acc.@1: 53.3 -> 54.9 (SLA); 53.3 -> 56.5 (P-SLA) | Helped |
| Filter noisy strokes with SLA | Sketch drawing assistance pipeline · QMUL-Shoe-V2 | MOS: 28.6 -> 85.7 | Helped |
| Filter noisy strokes with P-SLA | Sketch drawing assistance pipeline · QMUL-Shoe-V2 | MOS: 28.6 -> 57.1 | Helped |
| Filter noisy strokes with SLA or P-SLA | Sketch drawing assistance pipeline · QMUL-Chair-V2 | MOS: 35.8 -> 71.4 (SLA); 35.8 -> 57.1 (P-SLA) | Helped |
| Use Bhunia et al.'s trained filtering method instead of post-hoc SLA filtering | Pre-trained SBIR model · QMUL-Shoe-V2 | Acc.@1: 36.1 -> 43.7 | Helped |
| Use Bhunia et al.'s trained filtering method instead of post-hoc P-SLA filtering | Pre-trained SBIR model · QMUL-Chair-V2 | Acc.@1: 56.5 -> 64.8 | Helped |
| Remove a small stroke or selected points using an attribution-guided attack | ResNet-18 sketch classifier · QuickDraw, ϵ = 5 | classification accuracy: 67.2 -> 65.7 (SLA); 67.2 -> 65.1 (P-SLA) | Hurt |
| Remove a small stroke or selected points using an attribution-guided attack | ResNet-18 sketch classifier · TU-Berlin, ϵ = 15 | classification accuracy: 74.9 -> 68.5 (SLA); 74.9 -> 68.1 (P-SLA) | Hurt |
Surprising findings
- P-SLA-filtered sketches are preferred only 57.1% vs 42.9% over ground truth in the human study because filtering breaks strokes, and the post-hoc method trails the RL-trained baseline by about 7 points.
Limitations
- Partial stroke (P-SLA) filtering breaks lines and degrades visual quality of sketches.
- Post-hoc approach trails RL-trained baseline on noisy stroke removal accuracy.
- Human studies use only 7 participants with AI backgrounds.
Experimental setup
Datasets: TU-Berlin Sketchy QuickDraw QMUL-Shoe-V2 QMUL-Chair-V2
Benchmarks: TU-Berlin Sketchy QuickDraw QMUL-Shoe-V2 QMUL-Chair-V2
Metrics: mAP P@200 Acc.@1 Acc.@5 MOS classification accuracy drop human MCQ accuracy stroke order correlation
Baselines: Bhunia et al. (2023) RL-based stroke selection Sketch-A-Net
Builds on: Grad-CAM Gumbel-Softmax CLIP Sketch-A-Net
Closest papers by meaning
- SketchINR: A First Look into Sketches as Implicit Neural Representations (CVPR 2024 · similarity 0.86)
- It's All About Your Sketch: Democratising Sketch Control in Diffusion Models (CVPR 2024 · similarity 0.86)
- Enhance Sketch Recognition’s Explainability via Semantic Component-Level Parsing (AAAI 2024 · similarity 0.86)
- Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch (CVPR 2025 · similarity 0.84)
- Latent Representation Matters: Human-like Sketches in One-shot Drawing Tasks (NeurIPS 2024 · similarity 0.84)
- SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.