Research map / Label-efficient and robust learning / Recognition and long-tail

How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?

CVPR 2024 · Label-efficient and robust learning › Recognition and long-tail · approach: Correspondence and causal representation learning · 34 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an abstraction-aware fine-grained sketch-based image retrieval model that uses a k×d feature matrix embedding regularised by a pre-trained StyleGAN, an abstraction-level selector, and a differentiable Acc.@q surrogate loss.

Problem

Fine-grained sketch-based image retrieval performs poorly on sketches with varying levels of abstraction, since prior methods handle only sub-factors like style or stroke order and standard triplet losses treat all sketches equally.

Approach

Model sketch abstraction as a whole: a feature matrix embedding whose number of active rows is chosen by an abstraction identification head, with rows aligned to StyleGAN's coarse-to-fine latent groups, and a differentiable Acc.@q loss whose q is set by abstraction level (larger q for more abstract sketches).

Key results

• On ShoeV2 the proposed method reaches Acc.@1 45.3% and Acc.@5 77.3%, versus 39.9% for Partial-OT and 28.7% for Triplet-SN; on ChairV2 it reaches 72.1%.
• In the forensic sketch-photo setting it achieves Acc.@1 91.39%, above RGM's 88.94%.
• In early retrieval on ShoeV2 it reports m@A 86.22 versus 85.38 for On-the-fly.

Measured effects reported in the paper

Surprising findings

• Fine-tuning the StyleGAN instead of keeping it frozen lowers Acc.@1 to 30.2% on ShoeV2, which the authors attribute to loss of latent disentanglement.
• Generative image-translation baselines (B-pix2pix, B-CycleGAN) underperform the triplet baseline Triplet-SN.

Limitations

• Abstraction levels are defined by a heuristic mapping of 30%, 60%, and 100% rendering completion to coarse, medium, and fine, rather than learned from ground truth.
• Acc.@q is computed within a batch rather than over the full gallery, as a practical approximation.
• Evaluation is on small FG-SBIR datasets (ShoeV2, ChairV2) with a human study on a limited participant pool.

Experimental setup

Datasets: QMUL-ShoeV2 QMUL-ChairV2 UT Zappos50K FFHQ IIIT-D viewed sketch

Benchmarks: QMUL-ShoeV2 QMUL-ChairV2 IIIT-D viewed sketch

Metrics: Acc.@1 Acc.@5 Acc.@10 MOS ranking percentile VR@FAR=1%

Baselines: Triplet-SN HOLEF-SN B-pSp B-pix2pix B-CycleGAN Partial-OT CrossHier StyleMeUp On-the-fly SketchPVT SoDeep SmoothAP

Builds on: Triplet loss FG-SBIR (Sketch Me That Shoe) StyleGAN2 Smooth-AP Gumbel-Softmax VGG-16

Closest papers by meaning

• Picture That Sketch: Photorealistic Image Generation From Abstract Sketches (CVPR 2023 · similarity 0.90)
• Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch (CVPR 2025 · similarity 0.89)
• SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models (CVPR 2025 · similarity 0.88)
• It's All About Your Sketch: Democratising Sketch Control in Diffusion Models (CVPR 2024 · similarity 0.88)
• You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval (CVPR 2024 · similarity 0.87)
• ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling (AAAI 2025 · similarity 0.87)