Research map / Vision-language and multimodal / Human and embodied modeling
Exploring the Potential of Large Vision-Language Models for Unsupervised Text-Based Person Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses multiple LVLMs with coarse-to-fine prompts to caption unlabeled pedestrian images, then trains a BLIP-based text-based person retriever with uncertainty-guided filtering, Gaussian caption modeling, and multi-grained consistency distillation.
Problem
Text-based person retrieval normally requires manually annotated image-text pairs; the paper targets unsupervised retrieval that uses only images, with LVLM-generated captions that are noisy and uniform in style.
Approach
Generate n×g captions per image with several LVLMs under granularity-controlled prompts, filter noisy pairs with GMM-based cleanliness scores from similarity and matching scores, model caption features as multivariate Gaussians, and align multi-grained captions with KL-based ensemble consistency.
Key results
- On CUHK-PEDES, MUMA reaches 59.52% Rank-1 and 52.75% mAP in the unsupervised setting, beating GTR (47.53% R-1) by 11.99 points.
- On ICFG-PEDES, MUMA achieves 38.11% Rank-1 and 19.02% mAP, versus 31.58% R-1 for BLIP trained on LLaVA-1.5 captions.
- On RSTPReid, MUMA reaches 54.35% Rank-1 and 40.50% mAP, outperforming GTR's 45.60% Rank-1 by a wide margin.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MUMA rather than BLIP⋆ | BLIP-based unsupervised retrieval · CUHK-PEDES | R-1: +8.91% in Rank-1 accuracy | Helped |
| use MUMA rather than BLIP⋆ | BLIP-based unsupervised retrieval · CUHK-PEDES | mAP: +8.46% in mAP | Helped |
| use MUMA rather than BLIP⋆ | BLIP-based unsupervised retrieval · ICFG-PEDES | R-1: +6.53% in Rank-1 accuracy | Helped |
| use MUMA rather than BLIP⋆ | BLIP-based unsupervised retrieval · ICFG-PEDES | mAP: +5.82% in mAP | Helped |
| use MUMA rather than GTR | unsupervised retrieval · RSTPReid | R-1: 8.75% | Helped |
| use MUMA rather than BLIP⋆ | BLIP-based unsupervised retrieval · RSTPReid | mAP: 7.2% | Helped |
| use multiple LVLMs instead of single LLaVA-1.5 captions (ablation) | BLIP-based retrieval · CUHK-PEDES | R-1: 1.24% | Helped |
| use multiple LVLMs instead of single LLaVA-1.5 captions (ablation) | BLIP-based retrieval · CUHK-PEDES | mAP: 2.29% | Helped |
| use multi-grained prompts and captions (ablation) | BLIP-based retrieval · CUHK-PEDES | R-1: 2.26% | Helped |
| use multi-grained prompts and captions (ablation) | BLIP-based retrieval · CUHK-PEDES | mAP: 3.29% | Helped |
| add uncertainty-guided sample filtration (ablation) | MUMA with multi-model and multi-grained captions · CUHK-PEDES | R-1: 1.57% | Helped |
| add uncertainty-guided sample filtration (ablation) | MUMA with multi-model and multi-grained captions · CUHK-PEDES | mAP: 1.89% | Helped |
Surprising findings
- Using multiple LVLMs and multi-grained prompts gives gains over the single-LLaVA baseline, while the individual LVLMs perform comparably to each other.
Limitations
- Generated captions inherently contain noise, which the filtration module only partially removes.
- Performance remains well below fully supervised methods (e.g., CUHK-PEDES R-1 of 59.52 vs. 77.23 for AUL).
Experimental setup
Datasets: CUHK-PEDES ICFG-PEDES RSTPReid
Benchmarks: CUHK-PEDES ICFG-PEDES RSTPReid
Metrics: Rank-1 Rank-5 Rank-10 mAP
Baselines: GTR IRRA (unsupervised, LLaVA-1.5 captions) BLIP (LLaVA-1.5 captions) UniPT attribute-based captioning MAN
Builds on: BLIP IRRA GTR UniPT
Closest papers by meaning
- Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval (AAAI 2024 · similarity 0.91)
- Automatic Synthetic Data and Fine-grained Adaptive Feature Alignment for Composed Person Retrieval (NeurIPS 2025 · similarity 0.86)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.86)
- Modeling Thousands of Human Annotators for Generalizable Text-to-Image Person Re-identification (CVPR 2025 · similarity 0.86)
- Unifying Multi-Modal Uncertainty Modeling and Semantic Alignment for Text-to-Image Person Re-identification (AAAI 2024 · similarity 0.86)
- An Empirical Study of CLIP for Text-Based Person Search (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.