Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Asymmetric Mutual Alignment for Unsupervised Zero-Shot Sketch-Based Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AMA, an unsupervised zero-shot sketch-based image retrieval method combining DINO-style self-distillation with teacher-guided contrastive clustering, semantic similarity guidance, and cross-modality mutual alignment, using asymmetric EMA teacher updates.
Problem
Zero-shot sketch-based image retrieval without any labels for training data, where training and test categories do not overlap (UZS-SBIR).
Approach
Extends DINO-style self-distillation to separate sketch and image domains and adds a cross-modality mutual alignment module, where a stable EMA teacher guides contrastive clustering and semantic similarity learning, and inter-instance distance structures are aligned across sketch and image domains.
Key results
- AMA reaches 0.548 mAP on Sketchy Ext1 versus 0.457 for SD-Base and at most 0.198 for the other unsupervised baselines.
- On TU-Berlin Ext, AMA achieves 0.429 mAP, compared with 0.353 for SD-Base.
- On Quickdraw, AMA improves mAP to 0.112 from 0.101 for SD-Base (reported 5.5% gain).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add self-distillation loss (LSD) (ablation) | AMA · Sketchy Ext1 | mAP: 0.165 -> 0.548 | Helped |
| Add self-distillation loss (LSD) (ablation) | AMA · Sketchy Ext2 | mAP: 0.191 -> 0.507 | Helped |
| Add self-distillation loss (LSD) (ablation) | AMA · TU-Berlin Ext | mAP: 0.146 -> 0.429 | Helped |
| Add teacher-guided contrastive clustering loss (LCC) (ablation) | AMA · Sketchy Ext1 | mAP: 0.498 -> 0.548 | Helped |
| Add teacher-guided contrastive clustering loss (LCC) (ablation) | AMA · Sketchy Ext2 | mAP: 0.411 -> 0.507 | Helped |
| Add teacher-guided contrastive clustering loss (LCC) (ablation) | AMA · TU-Berlin Ext | mAP: 0.372 -> 0.429 | Helped |
| Add semantic similarity guidance loss (LSG) (ablation) | AMA · Sketchy Ext1 | mAP: 0.506 -> 0.548 | Helped |
| Add semantic similarity guidance loss (LSG) (ablation) | AMA · Sketchy Ext2 | mAP: 0.458 -> 0.507 | Helped |
| Add semantic similarity guidance loss (LSG) (ablation) | AMA · TU-Berlin Ext | mAP: 0.394 -> 0.429 | Helped |
| Add mutual alignment loss (LMA) (ablation) | AMA · Sketchy Ext1 | mAP: 0.493 -> 0.548 | Helped |
| Add mutual alignment loss (LMA) (ablation) | AMA · Sketchy Ext2 | mAP: 0.434 -> 0.507 | Helped |
| Add mutual alignment loss (LMA) (ablation) | AMA · TU-Berlin Ext | mAP: 0.346 -> 0.429 | Helped |
Surprising findings
- Removing the self-distillation loss drops Sketchy Ext1 mAP to 0.165, far below the full model, showing the foundation role of self-distillation.
Limitations
- Sketches and images of the same category remain only relatively close rather than fully fused in the embedding space.
- Retrieval errors occur when query sketches resemble non-matching images in shape and fine detail.
Experimental setup
Datasets: Sketchy Ext TU-Berlin Ext Quickdraw ImageNet
Benchmarks: Sketchy Ext1 Sketchy Ext2 TU-Berlin Ext Quickdraw
Metrics: mAP mAP@200 Prec@100 Prec@200
Baselines: MoCo v2 DINO CDS PCS FRL SD-Base
Builds on: DINO MoCo v2 relaxed metric loss
Closest papers by meaning
- Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (CVPR 2019 · similarity 0.92)
- Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval (CVPR 2019 · similarity 0.91)
- ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling (AAAI 2025 · similarity 0.91)
- Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR (CVPR 2023 · similarity 0.91)
- CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not (CVPR 2023 · similarity 0.91)
- Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.