Research map / Vision-language and multimodal / Cross-modal learning
Optimal Transport-based Labor-free Text Prompt Modeling for Sketch Re-identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OLTM, a sketch re-ID network that generates text attributes with a VQA model, injects them via learnable prompts and optimal transport into CLIP-based coarse and fine alignment, and adds a triplet assignment loss.
Problem
Sketch re-identification matches person images to sketch queries under a large modality gap; existing text-guided methods need costly manual annotations and use only global text cues.
Approach
Generate attribute text automatically with a pre-trained VQA model, combine it with learnable prompts, and use optimal transport to derive a local consensus that guides fine-grained sketch-image interaction; text is used only during training. A triplet assignment loss adds distribution-aware distance via OT.
Key results
- On Market-Sketch-1K multi-query, OLTM reaches 62.55% mAP and 69.48% Rank-1, versus 55.18% mAP and 56.63% Rank-1 for UNIReID.
- On PKU-Sketch, OLTM reaches 91.4% mAP and 94.0% Rank-1, above the best reported competitor (DALNet 86.2 mAP, CCSC/UNIReID-style baselines at 88.7 mAP for reproduced UNIReID).
- Ablation: VQA-generated attributes with learnable prompts improve mAP by 7.08% over direct alignment without text guidance; TCA adds 3.36% mAP over baseline and CFI adds 1.45%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OLTM instead of UNIReID | CLIP sketch Re-ID · Market-Sketch-1K single-query | mAP: 34.97 -> 38.35 | Helped |
| Use OLTM instead of UNIReID | CLIP sketch Re-ID · Market-Sketch-1K multi-query | Rank@1: 56.63 -> 69.48 | Helped |
| Use OLTM instead of the strongest reported baseline | Sketch Re-ID · PKU-Sketch | mAP: 88.7 -> 91.4 | Helped |
| Add VQA-generated text prompt reasoning to direct modality alignment (ablation) | OLTM · Market-Sketch-1K multi-query | mAP: +7.08% | Helped |
| Add learnable prompts rather than using fixed templates (ablation) | Text prompt reasoning · Market-Sketch-1K multi-query | mAP: +0.79% | Helped |
| Add text-injected coarse-grained alignment (TCA) (ablation) | OLTM baseline · Market-Sketch-1K multi-query | mAP: +3.36% | Helped |
| Add consensus-guided fine-grained interaction (CFI) (ablation) | OLTM with TCA · Market-Sketch-1K multi-query | mAP: +1.45% | Helped |
| Add triplet assignment loss instead of hard triplet loss (ablation) | OLTM · Market-Sketch-1K multi-query | Rank@1: 66.06 -> 69.48 | Helped |
| Replace simple multi-sketch fusion with average pooling (ablation) | Multi-query OLTM · Market-Sketch-1K multi-query | mAP: 62.55 -> 60.95 | Hurt |
| Replace simple multi-sketch fusion with non-local attention (ablation) | Multi-query OLTM · Market-Sketch-1K multi-query | mAP: 62.55 -> 60.98 | Hurt |
| Use BLIP or GIT instead of VILT for VQA attribute generation (ablation) | OLTM · Market-Sketch-1K multi-query | mAP: 62.55 (VILT); 62.63 (BLIP); 62.33 (GIT) | No clear effect |
| Add background attributes generated by VQA (ablation) | OLTM · Market-Sketch-1K | mAP: -1.62 | Hurt |
Surprising findings
- Adding background attributes (from a background VQA question) decreased Rank-1 by 2.81 and mAP by 1.62, attributed to sketches lacking background information.
- Text attributes are used only in training; inference uses only global CLS features, so no text cost at test time.
Limitations
- Performance degrades to near-random when sketches are too vague for humans to match.
- Quality of matching depends heavily on sketch quality.
- Adding background attributes hurt performance because sketches lack background information.
Experimental setup
Datasets: PKU-Sketch Market-Sketch-1K SYSU-MM01 RegDB
Benchmarks: Market-Sketch-1K PKU-Sketch SYSU-MM01 RegDB
Metrics: mAP Rank-1 Rank-5 Rank-10
Baselines: DDAG CM-NAS CAJ MMN DART DCLNet DSCNet DEEN BDG UNIReID SKetchTrans CCSC
Builds on: CLIP Sinkhorn-Knopp UNIReID BDG identity loss
Closest papers by meaning
- Differentiable Auxiliary Learning for Sketch Re-Identification (AAAI 2024 · similarity 0.88)
- ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling (AAAI 2025 · similarity 0.85)
- Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers (CVPR 2024 · similarity 0.85)
- How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? (CVPR 2024 · similarity 0.84)
- You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval (CVPR 2024 · similarity 0.84)
- A Pedestrian is Worth One Prompt: Towards Language Guidance Person Re-Identification (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.