Research map / Architectures and efficiency / Pruning and efficient tuning
Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SketchyNetV1 (cross-modal relational knowledge distillation into MobileNetV2) and SketchyNetV2 (RL-trained canvas-size selector over vector sketches) to make fine-grained sketch-based image retrieval efficient, cutting FLOPs by 99.37% with near-full accuracy.
Problem
Efficient inference for sketch data: lightweight photo networks such as MobileNetV2 lose substantial accuracy on fine-grained sketch-based image retrieval (FG-SBIR), and sketches' sparsity and abstraction are not exploited for compute savings.
Approach
A cross-modal relational knowledge distillation loss that matches sketch-photo pairwise Euclidean distances between a large teacher and a MobileNetV2 student, plus a GRU-based policy network that selects a per-sketch rasterisation canvas size from the vector sketch, trained with policy gradient using FLOPs and retrieval-rank rewards.
Key results
- SketchyNetV2 (MobileNetV2 student + canvas selector) on ShoeV2 reaches 32.77% Top-1 at 0.254G FLOPs versus 33.03% for the VGG-16 teacher at 40.18G FLOPs (99.37% FLOPs reduction).
- SketchyNetV1 distillation reduces FLOPs to 0.833G and parameters to 2.22M, with ShoeV2 Top-1 of 28.46% vs 28.71% for the Triplet-SN teacher at 256x256.
- The proposed relational KD outperforms regression, attention, and PKT KD baselines on cross-modal retrieval (e.g., ShoeV2 Top-1 of 26.x vs 22.82-24.31 for baselines, per Table 2 at the same student budget).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill teacher pairwise distances into a MobileNetV2 student | FG-SBIR on QMUL-ShoeV2 · QMUL-ShoeV2 | Top1 (%): 20.85 -> 28.46 | Helped |
| add an RL canvas-size selector to SketchyNetV1 | FG-SBIR on QMUL-ShoeV2 · QMUL-ShoeV2 | FLOPs (G): 0.833 -> 0.254 | Helped |
| add an RL canvas-size selector to SketchyNetV1 | FG-SBIR on QMUL-ShoeV2 · QMUL-ShoeV2 | Top1 (%): 28.46 -> 32.77 | No clear effect |
| remove the rank-1 reward from the canvas-selector objective (ablation) | SketchyNetV2 on QMUL-ShoeV2 · QMUL-ShoeV2 | Top1 (%): 32.77 -> 25.31 | Hurt |
| remove the triplet-loss reward from the canvas-selector objective (ablation) | SketchyNetV2 on QMUL-ShoeV2 · QMUL-ShoeV2 | Top1 (%): 32.77 -> 27.78 | Hurt |
| remove the FLOPs reward from the canvas-selector objective (ablation) | SketchyNetV2 on QMUL-ShoeV2 · QMUL-ShoeV2 | FLOPs (G): 0.254 -> 0.833 | Hurt |
| use MobileNetV2 instead of ResNet18 as the backbone (ablation) | SketchyNetV2 on QMUL-ShoeV2 · QMUL-ShoeV2 | Top1 (%): 26.72 -> 32.77 | Helped |
| use MobileNetV2 instead of EfficientNet as the backbone (ablation) | SketchyNetV2 on QMUL-ShoeV2 · QMUL-ShoeV2 | Top1 (%): 29.47 -> 32.77 | Helped |
| use vector input rather than full-resolution sketch-image input for the selector (ablation) | SketchyNetV2 on QMUL-ShoeV2 · QMUL-ShoeV2 | FLOPs (G): 5.730 -> 0.254 | Helped |
| use absolute-coordinate rather than offset-coordinate sketch vectors (ablation) | SketchyNetV2 on QMUL-ShoeV2 · QMUL-ShoeV2 | Top1 (%): 32.61 -> 32.77 | No clear effect |
| use GRU rather than LSTM as the sketch encoder (ablation) | SketchyNetV2 on QMUL-ShoeV2 · QMUL-ShoeV2 | Top1 (%): 31.41 -> 32.77 | Helped |
| use GRU rather than Transformer as the sketch encoder (ablation) | SketchyNetV2 on QMUL-ShoeV2 · QMUL-ShoeV2 | Top1 (%): 31.98 -> 32.77 | Helped |
Surprising findings
- Plain MobileNetV2 with triplet loss loses about 37% relative retrieval accuracy versus VGG-16, showing photo-efficient networks do not transfer directly to sketches.
- About 30% of query sketches that achieve perfect retrieval at full resolution still retrieve correctly when downsized to 32x32.
Limitations
- Gains depend on the quality of the teacher-trained critic; poorer retrieval models on ChairV2, Sketchy and FS-COCO increase FLOPs.
- Canvas-size selection is restricted to four discrete sizes, and the paper evaluates only FG-SBIR and a few related tasks.
- Efficiency results rely on FLOPs and a single GPU/phone measurement rather than a broad hardware study.
Experimental setup
Datasets: QMUL-ShoeV2 QMUL-ChairV2 Sketchy FS-COCO Sketchy (ext) Quick, Draw!
Benchmarks: QMUL-ShoeV2 QMUL-ChairV2 Sketchy FS-COCO Sketchy (ext) Quick, Draw!
Metrics: Acc@1 Acc@10 FLOPs parameter count inference time
Baselines: Triplet-SN HOLEF-SN Triplet-RL StyleVAE Partial-OT B-Regress B-AKD B-PKT B-BFR B-DRS B-Crop B-ThiNet
Builds on: Triplet-SN (Siamese triplet network for FG-SBIR) relational knowledge distillation policy gradient (REINFORCE) MobileNetV2 Sketch-a-Net
Closest papers by meaning
- Picture That Sketch: Photorealistic Image Generation From Abstract Sketches (CVPR 2023 · similarity 0.91)
- Sketch Me That Shoe (CVPR 2016 · similarity 0.91)
- It's All About Your Sketch: Democratising Sketch Control in Diffusion Models (CVPR 2024 · similarity 0.89)
- How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? (CVPR 2024 · similarity 0.89)
- SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models (CVPR 2025 · similarity 0.88)
- SketchINR: A First Look into Sketches as Implicit Neural Representations (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.