Research map / Vision-language and multimodal / Human and embodied modeling
Modeling Thousands of Human Annotators for Generalizable Text-to-Image Person Re-identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Human Annotator Modeling (HAM), which clusters style features of human captions and learns one prompt per cluster to make an MLLM mimic many annotator styles, plus Uniform Prototype Sampling (UPS) for diverse clusters, to auto-annotate the 1.0M-image HAM-PEDES dataset for text-to-image ReID pre-training.
Problem
Automatic captioning of pedestrian images with MLLMs yields captions with limited description-style diversity, which hurts the generalization of text-to-image person re-identification models trained on the generated data.
Approach
Cluster style features of human-written captions (after identity words are replaced by vague ones) and learn one soft prompt per cluster with the MLLM's adaptor, so the MLLM mimics many annotator styles; uniformly sample cluster centers in a defined style feature space (UPS) to cover wider style ranges and combine with KMeans clusters.
Key results
- On the direct transfer setting, HAM+KMeans+UPS with LLaVA1.6 reaches 60.60% Rank-1 on CUHK-PEDES versus 47.06% for fine-tuning the adaptor with 6.8K templates, and 36.87%/51.75% on ICFG/RSTPReid Rank-1 vs 26.74%/36.60% for the same baseline.
- Pretraining on 1.0M HAM-PEDES images gives direct-transfer Rank-1 of 70.15% on CUHK-PEDES, 59.63% on ICFG-PEDES, and 58.85% on RSTPReid, outperforming LUPerson-MLLM (57.61/38.36/51.50).
- With IRRA fine-tuning, HAM-PEDES pretraining reaches 77.71% Rank-1 on CUHK-PEDES; adding RDE fine-tuning gives 77.99%, RDE improved by 2.05%, 2.27%, and 7.20% Rank-1 on the three benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train on HAM-PEDES (1.0M) rather than the strongest listed baseline dataset | text-to-image ReID model in direct transfer · CUHK-PEDES | Rank-1: 57.61 -> 70.15 | Helped |
| pre-train on HAM-PEDES (1.0M) rather than the strongest listed baseline dataset | text-to-image ReID model in direct transfer · ICFG-PEDES | Rank-1: 57.08 -> 59.63 | Helped |
| pre-train on HAM-PEDES (1.0M) rather than the strongest listed baseline dataset | text-to-image ReID model in direct transfer · RSTPReid | Rank-1: 51.50 -> 58.85 | Helped |
| pre-train on HAM-PEDES and fine-tune with RDE rather than use the strongest listed prior method | text-to-image ReID model · CUHK-PEDES | Rank-1: 77.23 -> 77.99 | Helped |
| pre-train on HAM-PEDES and fine-tune with RDE rather than use the strongest listed prior method | text-to-image ReID model · ICFG-PEDES | Rank-1: 69.16 -> 69.95 | Helped |
| pre-train on HAM-PEDES and fine-tune with RDE rather than use the strongest listed prior method | text-to-image ReID model · RSTPReid | Rank-1: 71.65 -> 72.50 | Helped |
| add learned human style prompts and KMeans clustering rather than use adaptor fine-tuning with templates (ablation) | HAM caption generation with Qwen-VL-Chat-7B · CUHK-PEDES | Rank-1: 42.99 -> 52.19 | Helped |
| use KMeans clustering for style assignment rather than random sample selection (ablation) | HAM caption generation with Qwen-VL-Chat-7B · CUHK-PEDES | Rank-1: 43.60 -> 52.19 | Helped |
| use UPS rather than KMeans clustering for style prompts (ablation) | HAM caption generation with Qwen-VL-Chat-7B · CUHK-PEDES | Rank-1: 52.19 -> 54.53 | Helped |
| add UPS alongside DBSCAN rather than use DBSCAN alone (ablation) | HAM caption generation with Qwen-VL-Chat-7B · CUHK-PEDES | Rank-1: 46.93 -> 53.56 | Helped |
| combine KMeans and UPS rather than use KMeans alone at 2000 clusters (ablation) | HAM caption generation with Qwen-VL-Chat-7B · CUHK-PEDES | Rank-1: 51.20 -> 54.72 | Helped |
| combine KMeans and UPS with 1000 clusters each rather than 2000 each (ablation) | HAM caption generation with Qwen-VL-Chat-7B · CUHK-PEDES | Rank-1: 54.72 -> 55.34 | Helped |
Surprising findings
- Scaling the number of hand-designed description templates from 46 to 6.8K yields only marginal ReID gains, suggesting template redundancy and loss of wording style information.
- DBSCAN clustering performs poorly for style modeling because it labels sparse style regions as noise.
Limitations
- Does not account for noise or hallucinations in MLLM-generated annotations.
- Ablations use a single caption per image on 0.1M images and a single MLLM for the main dataset.
Experimental setup
Datasets: CUHK-PEDES ICFG-PEDES RSTPReid SYNTH-PEDES
Benchmarks: CUHK-PEDES ICFG-PEDES RSTPReid
Metrics: Rank-1 Rank-5 Rank-10 mAP
Baselines: MALS LUPerson-T LUPerson-MLLM SYNTH-PEDES IRRA RDE AUL APTM
Builds on: IRRA LUPerson-MLLM CLIP KMeans
Closest papers by meaning
- Unifying Multi-Modal Uncertainty Modeling and Semantic Alignment for Text-to-Image Person Re-identification (AAAI 2024 · similarity 0.89)
- Multi-Prompts Learning with Cross-Modal Alignment for Attribute-Based Person Re-identification (AAAI 2024 · similarity 0.89)
- Human-centered Interactive Learning via MLLMs for Text-to-Image Person Re-identification (CVPR 2025 · similarity 0.88)
- ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model (NeurIPS 2025 · similarity 0.88)
- Instruct-ReID: A Multi-purpose Person Re-identification Task with Instructions (CVPR 2024 · similarity 0.87)
- Empowering Visible-Infrared Person Re-Identification with Large Foundation Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.