Research map / Vision-language and multimodal / CLIP and zero-shot
PLIP: Language-Image Pre-training for Person Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PLIP, a language-image pre-training framework for person representation learning with three identity- and attribute-aware pretext tasks, pre-trained on a newly synthesized image-text person dataset SYNTH-PEDES.
Problem
General language-image pre-training (e.g., CLIP) transfers poorly to person-centric tasks because it ignores fine-grained person attributes and identity-level correspondences.
Approach
Three person-specific pretext tasks (text-guided image colorization, image-guided attribute phrase prediction, and identity-based vision-language contrast) trained jointly on a large synthetically captioned person dataset, with contrast at the identity rather than instance level.
Key results
- PLIP with Swin-B reaches 75.36 R@1 on CUHK-PEDES and 66.17 R@1 on ICFG-PEDES in text-based re-ID with the CMPM/C baseline, versus 58.91 for CLIP ViT-B.
- Applying PLIP ResNet50 to PPLR improves unsupervised Market1501 and MSMT17 mAP by 5.1% and 14.7%; PLIP with ISE reaches 87.6 mAP on Market1501 and 46.4 on MSMT17.
- PLIP with GLCNet (ResNet50) reaches 96.3 mAP on CUHK-SYSU and 53.7 mAP on PRW, and PLIP Swin-B achieves 57.8 mAP on PRW end-to-end person search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train with PLIP rather than LUP-NL | PPLR · Market1501 (unsupervised person Re-ID) | mAP: 5.1% | Helped |
| pre-train with PLIP rather than LUP-NL | PPLR · MSMT17 (unsupervised person Re-ID) | mAP: 14.7% | Helped |
| add text-guided image colorization (TIC) (ablation) | PLIP · CUHK-PEDES (zero-shot; ablation subset) | R@1: 30.2 -> 32.5 | Helped |
| add image-guided attributes prediction (IAP) (ablation) | PLIP · CUHK-PEDES (zero-shot; ablation subset) | R@1: 31.4 -> 32.5 | Helped |
| add identity-based vision-language contrast (IVLC) (ablation) | PLIP · CUHK-PEDES (zero-shot; ablation subset) | R@1 | Helped |
| add text-guided image colorization (TIC) (ablation) | PLIP · Market1501 (zero-shot; ablation subset) | R@1: 62.3 -> 63.1 | Helped |
| add image-guided attributes prediction (IAP) (ablation) | PLIP · Market1501 (zero-shot; ablation subset) | R@1: 62.9 -> 63.1 | Helped |
| add identity-based vision-language contrast (IVLC) (ablation) | PLIP · Market1501 (zero-shot; ablation subset) | R@1: 39.3 -> 63.1 | Helped |
| increase textual-description diversity (ablation) | SYNTH-PEDES pre-training · downstream performance (setting not specified) | performance | Helped |
Surprising findings
- CLIP pre-training underperforms the ImageNet-supervised baseline on several downstream tasks, e.g., unsupervised Market1501 mAP 75.0 vs 81.5 baseline with PPLR and MSMT17 mAP 7.6 vs 31.4.
Limitations
- Does not explicitly capture local cross-modal correspondence, limiting fine-grained representation learning.
- Multiple forward passes across pretext tasks increase memory and compute cost.
- Pre-training uses person-containing internet video data, raising privacy and security concerns; release is gated.
Experimental setup
Datasets: LUPerson-NL LPW SYNTH-PEDES Market1501 DukeMTMC CUHK-PEDES ICFG-PEDES PETA PA100K CUHK-SYSU PRW LIP
Benchmarks: CUHK-PEDES ICFG-PEDES Market1501 DukeMTMC PETA PA100K CUHK-SYSU PRW LIP PASCAL-Person-Part MSMT17
Metrics: R@1 R@5 R@10 mAP F1 mIoU
Baselines: CLIP MoCov3 BLIP LUP LUP-NL SOLIDER SSAN LGUR PPLR ISE SeqNet GLCNet
Builds on: CLIP CMPM loss BERT MoCov2 SOLIDER
Closest papers by meaning
- Visual Persona: Foundation Model for Full-Body Human Customization (CVPR 2025 · similarity 0.86)
- Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis (CVPR 2024 · similarity 0.86)
- An Empirical Study of CLIP for Text-Based Person Search (AAAI 2024 · similarity 0.86)
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (CVPR 2024 · similarity 0.85)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.85)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.