Research map / Vision-language and multimodal / CLIP and zero-shot
An Empirical Study of CLIP for Text-Based Person Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of fine-tuning CLIP for text-based person search, combining training tricks, data augmentations, and loss functions into a simple TBPS-CLIP baseline that reaches strong results on three benchmarks without extra modules.
Problem
Text-based person search (retrieving person images from natural language descriptions) has mostly relied on unimodal backbones or sophisticated add-on modules, leaving the potential of CLIP with simple design choices under-explored.
Approach
Instead of adding task-specific modules, systematically revisit CLIP fine-tuning design choices for TBPS, namely image and text augmentations, training tricks, and VLP-inspired losses (self-supervision, multi-view supervision, reversed ITC, cyclic ITC), and stack the effective ones into a lightweight two-stream CLIP baseline.
Key results
- TBPS-CLIP (ViT-B/16) reaches 73.54% Rank-1 on CUHK-PEDES, versus 65.37% for the CLIP ViT-B/16 baseline and 73.38% for IRRA.
- TBPS-CLIP (ViT-B/16) reaches 65.05% Rank-1 on ICFG-PEDES and 61.95% on RSTPReid, outperforming IRRA on these two datasets.
- Plugging TBPS-CLIP into IRRA raises CUHK-PEDES Rank-1 from 73.38% to 74.97%, and training takes 5 epochs versus 60 for IRRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the four training tricks: global gradients, dropout, locked bottom layer, and soft label (ablation) | CLIP · CUHK-PEDES | Rank-1: 60.67 -> 64.34 | Helped |
| add RandomResizedCrop (ablation) | CLIP* · CUHK-PEDES | Rank-1: 64.34 -> 65.29 | Helped |
| add GaussianBlur (ablation) | CLIP* · CUHK-PEDES | Rank-1: 64.34 -> 55.30 | Hurt |
| add RandomHorizontalFlip (ablation) | CLIP* · CUHK-PEDES | Rank-1: 64.34 -> 65.48 | Helped |
| add the Augmentation Pool of two randomly selected effective image augmentations (ablation) | CLIP* · CUHK-PEDES | Rank-1: 64.34 -> 66.13 | Helped |
| add Random Deletion (ablation) | CLIP* · CUHK-PEDES | Rank-1: 64.34 -> 65.53 | Helped |
| add Random Swap (ablation) | CLIP* · CUHK-PEDES | Rank-1: 64.34 -> 56.41 | Hurt |
| combine the selected image and text augmentations (ablation) | CLIP* · CUHK-PEDES | Rank-1: 64.34 -> 66.78 | Helped |
| add image self-supervision (ablation) | CLIP* + Aug · CUHK-PEDES | Rank-1: 66.78 -> 67.54 | Helped |
| add text self-supervision (ablation) | CLIP* + Aug · CUHK-PEDES | Rank-1: 66.78 -> 65.97 | Hurt |
| add reversed image-text contrastive loss (ablation) | CLIP* + Aug · CUHK-PEDES | Rank-1: 66.78 -> 68.19 | Helped |
| add all selected loss functions (ablation) | CLIP* + Aug · CUHK-PEDES | Rank-1: 66.78 -> 69.54 | Helped |
Surprising findings
- CoOp and CLIP-Adapter perform far worse than vanilla CLIP in 5% few-shot TBPS (about 11% vs 38% Rank-1), attributed to the gap between CLIP pre-training data and fine-grained person retrieval.
- GaussianBlur and RandomGrayscale behave differently than expected: RandomGrayscale improves results while GaussianBlur severely hurts them (55.30 vs 64.34 Rank-1).
Limitations
- Relies on common, previously known techniques (augmentations, losses) rather than a novel architecture or training objective.
- Few-shot and compression analyses are preliminary and limited to CUHK-PEDES in the main text.
Experimental setup
Datasets: CUHK-PEDES ICFG-PEDES RSTPReid
Benchmarks: CUHK-PEDES ICFG-PEDES RSTPReid
Metrics: Rank-1 Rank-5 Rank-10 mAP mINP
Baselines: IRRA RaSa CFine TP-TPS CoOp CLIP-Adapter LGUR SSAN
Builds on: CLIP ALBEF CyCLIP DeCLIP EDA
Closest papers by meaning
- TF-CLIP: Learning Text-Free CLIP for Video-Based Person Re-identification (AAAI 2024 · similarity 0.88)
- DM-Adapter: Domain-Aware Mixture-of-Adapters for Text-Based Person Retrieval (AAAI 2025 · similarity 0.87)
- Chat-based Person Retrieval via Dialogue-Refined Cross-Modal Alignment (CVPR 2025 · similarity 0.87)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.86)
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.86)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.