Research map / Label-efficient and robust learning / Recognition and long-tail
Enhance Sketch Recognition’s Explainability via Semantic Component-Level Parsing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Semantic Component-level Memory (SCM) module embedded in a Structured Sketch Recognition (SSR) network that parses strokes into learnable semantic-component keys, enabling explainable sketch recognition with or without stroke-level component labels.
Problem
Sketch recognition networks give category predictions without interpretable reasons; existing explainability methods (activation maps, counterfactuals) are ill-suited to stroke-based sketches.
Approach
A learnable multi-head memory of semantic-component keys compares stroke features to keys via a Student-t-like kernel, producing stroke- or component-level features fed to a Transformer; supervision comes either from stroke-level component labels or only from category-level component existence (compositionality), so recognition is explained as being composed of the components constituting the clas
Key results
- On SPG, SSR with F_s=Q reaches 96.67% Acc@1 and SSR with F_s as Eq.(2) reaches 90.12 C-Metric, while SketchRecSeg is best overall at 97.47 Acc@1 and 91.65 C-Metric.
- On SketchIME, SSR with F_s=Q achieves 91.48% Acc@1 and 94.91 C-Metric, outperforming SketchGNN (94.01 C-Metric) and ResNet18 (89.01 Acc@1).
- Using prior component-existence supervision (L6) raises recognition accuracy from 92.02% to 94.04% (F_c as Eq.3) and from 90.71% to 94.55% (F_c=C^T*Q) on SPG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add compositionality loss L6 (ablation) | SSR with Eq. (3) component-level features · SPG, C-Labels and Prior Info | Acc@1: 92.02% vs. 94.04% | Helped |
| add compositionality loss L6 (ablation) | SSR with C^T*Q component features · SPG, C-Labels and Prior Info | Acc@1: 90.71% vs. 94.55% | Helped |
| replace Eq. (2) stroke-level fusion with C*K features (ablation) | SSR · SPG, C-Labels and SC-Labels | Acc@1: 95.81 -> 96.62 | Helped |
| replace Eq. (2) stroke-level fusion with C*K features (ablation) | SSR · SPG, C-Labels and SC-Labels | C-Metric: 90.12 -> 89.69 | Hurt |
| feed Q features to the Transformer instead of Eq. (2) fused features (ablation) | SSR · SPG, C-Labels and SC-Labels | Acc@1: 95.81 -> 96.67 | Helped |
| feed Q features to the Transformer instead of Eq. (2) fused features (ablation) | SSR · SPG, C-Labels and SC-Labels | C-Metric: 90.12 -> 89.42 | Hurt |
| replace Eq. (2) stroke-level fusion with C*K features (ablation) | SSR · SketchIME | Acc@1: 89.88 -> 87.92 | Hurt |
| replace Eq. (2) stroke-level fusion with C*K features (ablation) | SSR · SketchIME | C-Metric: 94.59 -> 94.43 | Hurt |
| use category labels only, without component-label supervision or prior information (ablation) | SSR with Eq. (2) stroke-level fusion · SPG | Acc@1: 95.81 -> 88.48 | Hurt |
| use category labels only and C*K features (ablation) | SSR · SPG | Acc@1: 96.62 -> 91.41 | Hurt |
| use category labels only and Q features (ablation) | SSR · SPG | Acc@1: 96.67 -> 91.01 | Hurt |
Surprising findings
- Without any stroke or component supervision, the memory keys were hard to learn and the category-only configurations performed notably worse (88-92% Acc@1 on SPG).
- The SketchRecSeg baseline, which uses image plus SVG inputs in a two-stream design, outperforms the proposed SVG-only SSR network on SPG.
Limitations
- On SPG the proposed SSR does not beat SketchRecSeg, which uses a two-stream image and SVG design.
- Memory keys learned without component labels are hard to learn, so performance drops substantially in the category-label-only setting.
- Explainability is argued qualitatively via visualizations rather than measured with a user or quantitative explanation metric.
Experimental setup
Datasets: SPG SketchIME
Benchmarks: SPG SketchIME
Metrics: Acc@1 C-Metric
Baselines: ViT BiGRU ResNet18 MGT SketchSegNet SketchGNN SketchRecSeg
Builds on: ViT EdgeConv MemGNN SketchGNN SketchXAI
Closest papers by meaning
- Open Vocabulary Semantic Scene Sketch Understanding (CVPR 2024 · similarity 0.88)
- How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? (CVPR 2024 · similarity 0.87)
- SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models (CVPR 2025 · similarity 0.86)
- What Sketch Explainability Really Means for Downstream Tasks? (CVPR 2024 · similarity 0.86)
- SketchINR: A First Look into Sketches as Implicit Neural Representations (CVPR 2024 · similarity 0.85)
- Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.