Research map / Image generation / Text-to-image diffusion
Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReID
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 1.0M-image LUPerson-MLLM caption dataset with MLLMs (Qwen-VL-Chat, Shikra) using ChatGPT-generated description templates, and trains CLIP-ViT-B/16 with a noise-aware masking method for transferable text-to-image person ReID.
Problem
Text-to-image person ReID is limited by scarce annotated text, and models trained on one dataset generalize poorly to others; MLLM-generated captions are low in diversity and contain noisy words.
Approach
Use template-guided MLLM captioning (templates produced by multi-turn ChatGPT dialogue) to diversify descriptions, and Noise-Aware Masking that scores each text token by its max patch-similarity to the image and masks likely-wrong words in the next epoch without predicting them.
Key results
- In direct transfer, training on LUPerson-MLLM (1.0M) reaches 57.61 Rank-1 on CUHK-PEDES, 38.36 on ICFG-PEDES and 51.50 on RSTPReid, versus 12.65/6.67/13.45 for original CLIP.
- NAM improves direct-transfer Rank-1 by 2.32%, 3.49% and 2.05% over equal-probability masking on the three benchmarks (0.1M ablation setting).
- With pretrained weights plus IRRA, Rank-1 on RSTPReid improves to 68.50 (traditional setting), an 8.30-point gain over IRRA's 60.20.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dynamic caption templates (TDE) with static captions rather than the two static captions alone (ablation) | Transferable text-to-image ReID model · CUHK-PEDES direct transfer | Rank-1: 46.00 -> 50.32 | Helped |
| Use dynamic caption templates (TDE) with static captions rather than the two static captions alone (ablation) | Transferable text-to-image ReID model · ICFG-PEDES direct transfer | Rank-1: 26.74 -> 29.12 | Helped |
| Use dynamic caption templates (TDE) with static captions rather than the two static captions alone (ablation) | Transferable text-to-image ReID model · RSTPReid direct transfer | Rank-1: 41.10 -> 45.70 | Helped |
| Add noise-aware masking (NAM) (ablation) | TDE text-to-image ReID model · CUHK-PEDES direct transfer | Rank-1: 50.32 -> 52.64 | Helped |
| Add noise-aware masking (NAM) (ablation) | TDE text-to-image ReID model · ICFG-PEDES direct transfer | Rank-1: 29.12 -> 32.61 | Helped |
| Add noise-aware masking (NAM) (ablation) | TDE text-to-image ReID model · RSTPReid direct transfer | Rank-1: 45.70 -> 47.75 | Helped |
| Add MLM loss to NAM (ablation) | NAM text-to-image ReID model · CUHK-PEDES direct transfer | Rank-1: 52.64 -> 48.79 | Hurt |
| Add MLM loss to NAM (ablation) | NAM text-to-image ReID model · ICFG-PEDES direct transfer | Rank-1: 32.61 -> 27.36 | Hurt |
| Add MLM loss to NAM (ablation) | NAM text-to-image ReID model · RSTPReid direct transfer | Rank-1: 47.75 -> 44.45 | Hurt |
| Increase pretraining data from 0.1 M to 1.0 M images (ablation) | Proposed text-to-image ReID model · ICFG-PEDES direct transfer | Rank-1: 32.61 -> 38.36 | Helped |
Surprising findings
- Applying an MLM loss on top of NAM hurts performance (e.g., CUHK-PEDES Rank-1 drops from 52.64 to 48.79), which the authors attribute to noise in MLLM descriptions.
Limitations
- TDE effectiveness is limited by the number of sentence templates (46 used).
- NAM may occasionally fail to mask noisy tokens.
Experimental setup
Datasets: LUPerson CUHK-PEDES ICFG-PEDES RSTPReid
Benchmarks: CUHK-PEDES ICFG-PEDES RSTPReid
Metrics: Rank-1 Rank-5 Rank-10 mAP
Baselines: CLIP MALS LUPerson-T IRRA RaSa APTM UniPT LGUR
Builds on: CLIP SDM loss IRRA LUPerson
Closest papers by meaning
- Modeling Thousands of Human Annotators for Generalizable Text-to-Image Person Re-identification (CVPR 2025 · similarity 0.81)
- Human-centered Interactive Learning via MLLMs for Text-to-Image Person Re-identification (CVPR 2025 · similarity 0.80)
- MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning (CVPR 2026 · similarity 0.80)
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.80)
- SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding (ICLR 2025 · similarity 0.79)
- RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models (CVPR 2025 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.