Research map / Vision-language and multimodal / Human and embodied modeling
You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a CLIP-based sketch+text composed fine-grained image retrieval method that maps a sketch to a pseudo-word token and trains without paired fine-grained text, using a sketch-photo difference proxy and neutral-text regularisation.
Problem
Fine-grained image retrieval has relied on sketches alone; combining sketch with text is hard because fine-grained sketch-text-photo triplets are costly to collect and naive feature fusion distorts the composed semantics.
Approach
Convert the sketch into a CLIP pseudo-word token via a visual-to-word converter, compose it with text through CLIP's frozen text encoder, and train without captions by treating the sketch-photo difference embedding as a proxy for missing text (compositionality constraint), regularised with GPT-generated neutral phrases and a text-to-text prompt loss.
Key results
- On ShoeV2, the method reaches Acc.@5 of 47.3 and Acc.@10 of 79.1, versus 38.4/64.8 for SEARLE (best baseline).
- On FS-COCO scene-level retrieval, Acc.@5 is 22.7 versus 15.9-ish for prior SoTA SEARLE at 17.7 and TASK-former at 12.7.
- Ablation: removing the compositionality constraint drops ShoeV2 Acc.@5 from 47.3 to 32.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sketch+text composition with the proposed framework rather than the strongest baseline | Fine-grained composed retrieval · ShoeV2 | Acc.@5: 38.4 -> 47.3 | Helped |
| Use sketch+text composition with the proposed framework rather than the strongest baseline | Fine-grained composed retrieval · ChairV2 | Acc.@5: 60.8 -> 73.5 | Helped |
| Use sketch+text composition with the proposed framework rather than the strongest baseline | Fine-grained composed retrieval · FS-COCO | Acc.@5: 17.7 -> 22.7 | Helped |
| Add text-to-text generalisation loss LTT (ablation) | Proposed method · ShoeV2 | Acc.@5: 41.8 -> 47.3 | Helped |
| Add auxiliary reconstruction loss Lrec (ablation) | Proposed method · ShoeV2 | Acc.@5: 40.7 -> 47.3 | Helped |
| Add region-aware triplet loss LRT (ablation) | Proposed method · FS-COCO | Acc.@5: 10.5 -> 22.7 | Helped |
| Add the compositionality constraint (ablation) | Proposed method · ShoeV2 | Acc.@5: 32.5 -> 47.3 | Helped |
Surprising findings
- Pseudo-word token from a single sketch achieves 98.35% Acc.@1 sketch retrieval on ShoeV2 without any text, suggesting the token captures sketch content well.
Limitations
- Evaluation text captions for ShoeV2, ChairV2 and Sketchy were manually collected by the authors, so the evaluation depends on a small handcrafted set.
- Fine-grained sketch representation with one token may be sub-optimal; multiple tokens gave only marginal gains.
- Domain labels in ImageNet-R are noisy, so only four domains were used.
Experimental setup
Datasets: QMUL-ShoeV2 QMUL-ChairV2 Sketchy FS-COCO SketchyCOCO ImageNet-R
Benchmarks: QMUL-ShoeV2 QMUL-ChairV2 Sketchy FS-COCO SketchyCOCO ImageNet-R
Metrics: Acc.@5 Acc.@10 r@10 r@50 FID
Baselines: B-CLIP (S) B-DINOv2 (S) B-CLIP (T) B-CLIP (S+T) Combiner TASK-former SceneTrilogy Pic2Word SEARLE
Builds on: CLIP Pic2Word SEARLE Combiner CoOp
Closest papers by meaning
- Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions (AAAI 2024 · similarity 0.89)
- ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling (AAAI 2025 · similarity 0.88)
- Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers (CVPR 2024 · similarity 0.87)
- How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? (CVPR 2024 · similarity 0.87)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.86)
- Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.