Research map / Vision-language and multimodal / CLIP and zero-shot
A Pedestrian is Worth One Prompt: Towards Language Guidance Person Re-Identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptSG, an end-to-end CLIP-based person re-ID framework that inverts images into pseudo-word tokens and uses language prompts to guide patch cross-attention.
Problem
Person re-identification needs semantic part localization and identity-specific features, but existing CLIP-based ReID (CLIP-ReID) uses a two-stage pipeline with ID-specific soft prompts that cannot be used for unseen classes or at inference with text.
Approach
Use a lightweight inversion network to map each image's CLIP global embedding to a pseudo-token inserted into the prompt "A photo of a S* person", and use a cross-attention multimodal interaction module where text queries reweight patch tokens, trained end-to-end with ID, triplet and supervised contrastive losses.
Key results
- With ViT-B/16 on MSMT17, PromptSG reaches 68.5 mAP / 86.0 R-1 (ViT-B/16 CLIP-based group), versus CLIP-ReID's 63.0 / 84.4 under the same CLIP-based row of Table 2 (the paper reports over 10% gain on MSMT17 with its 87.2 mAP / 92.6 R-1 setting)
- On Market-1501 with ViT-B/16, PromptSG achieves 94.6 mAP and 97.0 R-1, versus CLIP-ReID's 89.6 mAP and 95.5 R-1.
- Training time is about 0.51x of CLIP-ReID on Market-1501 and 0.47x on MSMT17, roughly 2x faster.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PromptSG with CLIP ViT-B/16 instead of the strongest reported baseline | person ReID · Market-1501 | mAP: 90.2 -> 94.6 | Helped |
| use PromptSG with CLIP ViT-B/16 instead of the strongest reported baseline | person ReID · MSMT17 | mAP: 68.9 -> 87.2 | Helped |
| use PromptSG with CLIP ViT-B/16 instead of the strongest reported baseline | person ReID · DukeMTMC | mAP: 82.0 -> 81.6 | Hurt |
| use PromptSG with CLIP ViT-B/16 instead of the strongest reported baseline | person ReID · CUHK03-NP (labeled) | mAP: 83.0 -> 83.1 | Helped |
| remove image-to-text supervised contrastive loss (ablation) | PromptSG · MSMT17 | mAP: 87.2 -> 85.2 | Hurt |
| remove text-to-image supervised contrastive loss (ablation) | PromptSG · MSMT17 | mAP: 87.2 -> 84.5 | Hurt |
| remove the multimodal interaction module (ablation) | PromptSG · MSMT17 | mAP: 87.2 -> 71.4 | Hurt |
| train without composing the pseudo-token into the prompt (ablation) | PromptSG · MSMT17 | mAP: 85.3 -> 87.2 | Hurt |
| add one cross-attention layer (ablation) | PromptSG · Market-1501 | mAP: 89.4 -> 91.1 | Helped |
| add one self-attention layer after cross-attention (ablation) | PromptSG · Market-1501 | mAP: 91.1 -> 93.0 | Helped |
| use a simplified fixed prompt at inference instead of the composed prompt (ablation) | PromptSG · Market-1501 | mAP: 94.6 -> 94.1 | Hurt |
Surprising findings
- Replacing the composed prompt with the fixed simplified prompt 'A photo of a person' at inference keeps most of the performance while avoiding the text-encoder cost.
Limitations
- Fine-tuning the visual encoder for strong supervised performance may hurt zero-shot generalization, as the authors note.
- Composing the text prompt at inference roughly doubles inference cost (FPS 0.48x of baseline).
Experimental setup
Datasets: Market-1501 MSMT17 DukeMTMC CUHK03-NP
Benchmarks: Market-1501 MSMT17 DukeMTMC CUHK03-NP
Metrics: mAP Rank-1
Baselines: CLIP-ReID TransReID PHA DCAL AAformer ALDER LTReID ISP RGA-SC
Builds on: CLIP CLIP-ReID textual inversion CoOp TransReID
Closest papers by meaning
- Enhanced Visual-Semantic Interaction with Tailored Prompts for Pedestrian Attribute Recognition (CVPR 2025 · similarity 0.90)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.88)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.87)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Prompt-Driven Referring Image Segmentation with Instance Contrasting (CVPR 2024 · similarity 0.86)
- ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.