Research map / Vision-language and multimodal / Human and embodied modeling
Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the CSTBIR task (composite sketch+text image retrieval over natural scenes), a ~2M-query dataset, and STNet, a CLIP-based multimodal transformer trained with contrastive, classification, sketch-guided detection, and sketch reconstruction losses.
Problem
Retrieving natural scene images for composite queries combining a rough object sketch (for hard-to-name objects) with complementary text describing attributes or interactions; existing TBIR and SBIR methods cannot handle this combination.
Approach
A multimodal transformer built on pretrained CLIP text and ViT encoders, where the sketch embedding attends over image patch embeddings to localize the queried object, trained jointly with contrastive, object classification, sketch-guided detection (YOLO-style), and sketch reconstruction losses.
Key results
- STNet reaches R@10 of 73.7 on Test-1K and 38.7 on Test-5K, versus 67.0 and 34.8 for the two-stage baseline and 50.6 and 24.2 for CLIP.
- On the Open-Category set of 70 unseen classes, STNet achieves R@50 of 62.3 and MdR 27.5, compared with 48.8 R@50 for the two-stage model.
- With instance-level sketches on Test-1K, STNet obtains R@10 of 70.3 versus 61.4 for the two-stage model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the two-stage sketch-to-label then text retrieval approach with joint sketch-and-text STNet | CSTBIR retrieval · Test-1K | Recall@10: 67.0 -> 73.7 | Helped |
| replace the two-stage sketch-to-label then text retrieval approach with joint sketch-and-text STNet | CSTBIR retrieval · Test-5K | Recall@10: 34.8 -> 38.7 | Helped |
| replace the two-stage sketch-to-label then text retrieval approach with joint sketch-and-text STNet | CSTBIR retrieval · Open-Category | Recall@10: 29.0 -> 37.2 | Helped |
| replace the two-stage sketch-to-label then text retrieval approach with joint sketch-and-text STNet | CSTBIR retrieval · Test-1K instance-level sketches | Recall@10: 61.4 -> 70.3 | Helped |
| combine sketch and text instead of using text-only contrastive queries (ablation) | STNet · Test-1K | Recall@10: 50.6 -> 68.4 | Helped |
| add sketch-guided object detection loss (ablation) | STNet · Test-1K | Recall@10: 70.4 -> 73.7 | Helped |
| add sketch reconstruction loss (ablation) | STNet · Test-1K | Recall@10: 71.2 -> 73.7 | Helped |
| add text and image object classification losses (ablation) | STNet · Test-1K | Recall@10: 69.4 -> 73.7 | Helped |
Surprising findings
- The two-stage model using ChatGPT-generated or human object descriptions performs worse than the two-stage model using plain class names.
- Text-only CLIP outperforms the sketch-only ViT-Siamese model, indicating sketches carry less information than text in this rough-sketch setting.
Limitations
- Training sketches come from Quick, Draw! and are not truly difficult-to-name objects; the main dataset uses 258 object classes that overlap with Visual Genome.
- The Open-Category test set is small (750 queries, 1K gallery images) and only partially covers difficult-to-name objects.
- Two-stage baselines are restricted to closed-world object vocabularies, which the authors note limits their generalization.
Experimental setup
Datasets: Visual Genome Quick, Draw! ImageNet-21K
Benchmarks: CSTBIR Test-1K CSTBIR Test-5K CSTBIR Open-Category test set
Metrics: Recall@K Median Rank
Baselines: Doodle2Search DeepSBIR ViT-Siamese VisualBERT ViLT CLIP TIRG Taskformer Two-stage Two-stage (desc)
Builds on: CLIP YOLO TIRG Taskformer
Closest papers by meaning
- ST-Bridge: Bridging Sketch and Text with Large Language Models for Coarse-to-Fine Image Retrieval (NeurIPS 2026 · similarity 0.90)
- You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval (CVPR 2024 · similarity 0.89)
- Asymmetric Mutual Alignment for Unsupervised Zero-Shot Sketch-Based Image Retrieval (AAAI 2024 · similarity 0.85)
- ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling (AAAI 2025 · similarity 0.85)
- INQUIRE: A Natural World Text-to-Image Retrieval Benchmark (NeurIPS 2024 · similarity 0.85)
- How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.