Research map / Label-efficient and robust learning / Self-supervised and contrastive
ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARNet, a self-supervised FG-SBIR framework with dual weight-sharing ViT encoders, a contrastive objective for intra- and inter-sample alignment, and a Multi-Scale Token Recycling module that reuses discarded ViT patch tokens; also releases the Clothes-V1 fashion sketch-image dataset.
Problem
Fine-grained sketch-based image retrieval (FG-SBIR) struggles with scalability and training stability of triplet-loss-based methods, and standard ViT pipelines discard patch tokens that may hold useful information.
Approach
Replace triplet loss with a contrastive objective applied to both intra-sample (augmented vs. original) and inter-sample (sketch vs. image) pairs using dual weight-sharing encoders, and recycle discarded ViT patch tokens through a multi-scale transformer with a token-wise contrastive map loss.
Key results
- ARNet Full reaches 75.45% R@1 on QMUL-Chair-V2 versus 71.22% for EUPS-SBIR, the previous best.
- On QMUL-Shoe-V2, ARNet Full reaches 42.91% R@1, below EUPS-SBIR's 44.18%, placing third.
- On Clothes-V1, ARNet Full reaches 95.27% Acc.@1 versus 89.47% for EUPS-SBIR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MSTR token recycling to ARNet Basic (ablation) | ARNet on Clothes-V1 · Clothes-V1 | Acc.@1: 94.12 -> 95.27 | Helped |
| Add MSTR token recycling to ARNet Basic (ablation) | ARNet on QMUL-Chair-V2 · QMUL-Chair-V2 | R@1: 73.31 -> 75.45 | Helped |
| Add MSTR token recycling to ARNet Basic (ablation) | ARNet on QMUL-Shoe-V2 · QMUL-Shoe-V2 | R@1: 40.11 -> 42.91 | Helped |
| Add dual weight-sharing networks (ablation) | ARNet with VGG16 on QMUL-Chair-V2 · QMUL-Chair-V2 | Acc.@1: 22.42 -> 31.32 | Helped |
| Add inter-sample loss to dual weight-sharing networks (ablation) | ARNet with ViT-B/16 on QMUL-Chair-V2 · QMUL-Chair-V2 | Acc.@1: 58.36 -> 73.31 | Helped |
| Use A1 augmentation instead of A0 in the single-network setup (ablation) | ARNet single-network data loading · QMUL-Chair-V2 | Acc.@1: 59.43 -> 68.33 | Helped |
| Replace A1 with A2 in dual-network A0+A1 augmentation (ablation) | ARNet dual-network data loading · QMUL-Chair-V2 | Acc.@1: 73.31 -> 69.75 | Hurt |
| Use sketch-anchor triplet loss instead of contrastive loss (ablation) | ARNet with A0 data loading · QMUL-Chair-V2 | Acc.@1: 59.43 -> 57.65 | Hurt |
| Use image-anchor triplet loss instead of contrastive loss (ablation) | ARNet with A0 data loading · QMUL-Chair-V2 | Acc.@1: 59.43 -> 24.20 | Hurt |
| Replace MSTR token recycling with MaxPool (ablation) | ARNet MSTR on QMUL-Chair-V2 · QMUL-Chair-V2 | Acc.@1: 73.31 -> 72.95 | Hurt |
| Replace MSTR token recycling with AvgPool (ablation) | ARNet MSTR on QMUL-Chair-V2 · QMUL-Chair-V2 | Acc.@1: 73.31 -> 71.89 | Hurt |
| Replace MSTR token recycling with concatenation (ablation) | ARNet MSTR on QMUL-Chair-V2 · QMUL-Chair-V2 | Acc.@1: 73.31 -> 65.48 | Hurt |
Surprising findings
- Simple patch-token aggregation (MaxPool, AvgPool, concatenation) fails to improve and can hurt performance, while the learned MSTR module helps.
- Data augmentation A2 (MoCo-style) hurt performance in the single-network setting and was incompatible with the fine-grained datasets.
Limitations
- On QMUL-Shoe-V2 the method does not reach the top result, attributed to larger dataset size and irregular sketch strokes.
- Clothes-V1 is small (1200 sketches, 500 images), and the paper does not report variance across runs.
Experimental setup
Datasets: QMUL-Chair-V2 QMUL-Shoe-V2 Clothes-V1
Benchmarks: QMUL-Chair-V2 QMUL-Shoe-V2 Clothes-V1
Metrics: R@1 R@5 R@10 Acc.@1 Acc.@5 Acc.@10
Baselines: Triplet-SN Triplet-Att-SN B-Siamese OnTheFly EUPS-SBIR CMHM-SBIR SketchAA Semi-Sup StyleMeUp Adpt-SBIR Part-SBIR NT-SBIR
Builds on: ViT triplet loss MoCo contrastive learning
Closest papers by meaning
- Generalising Fine-Grained Sketch-Based Image Retrieval (CVPR 2019 · similarity 0.92)
- Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR (CVPR 2023 · similarity 0.92)
- Sketch Me That Shoe (CVPR 2016 · similarity 0.91)
- Asymmetric Mutual Alignment for Unsupervised Zero-Shot Sketch-Based Image Retrieval (AAAI 2024 · similarity 0.91)
- You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval (CVPR 2024 · similarity 0.88)
- How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.