Research map / Vision-language and multimodal / Cross-modal learning
Chat-based Person Retrieval via Dialogue-Refined Cross-Modal Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces chat-based person retrieval (ChatPR) with an LLM-generated dialogue dataset ChatPedes and the DiaNA dual-encoder framework using Llama 3 plus adaptive attribute refiners and random round retaining.
Problem
Text-based person retrieval uses a single-shot text query that often lacks complete information about the target person; no dialogue-image paired data exists to support multi-round conversational queries.
Approach
Replace single-shot text queries with multi-round dialogues, build a dialogue-image dataset by having LLMs generate questions and simulated answers from CUHK-PEDES captions with ensemble-vote cleaning, and align dialogues and images via learnable attribute queries (adaptive attribute refiners) with global and local losses plus random round retaining augmentation.
Key results
- DiaNA reaches 75.67% R-1 and 66.89% mAP on ChatPedes with data cleaning, versus 52.77% R-1 for the best baseline RDE.
- Random round retaining improves R-1 by 2.75 points at one dialogue round and 0.49 points on the full dialogue.
- Using both Llama 3 and InternVL dialogues gives 75.67% R-1 versus 71.62% with Llama 3 alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply data cleaning to generated dialogues (ablation) | DiaNA on ChatPedes · ChatPedes test | R-1: 73.72 -> 75.67 | Helped |
| combine dialogues generated by Llama 3 and InternVL instead of using either generator alone (ablation) | ChatPedes and DiaNA · ChatPedes test | R-1: 71.01; 71.62 -> 75.67 | Helped |
| add adaptive attribute refiner modules with attribute queries (K = 16) (ablation) | DiaNA · ChatPedes test | R-1 | Helped |
| add random round retaining (R3) during training (ablation) | DiaNA · ChatPedes test, dialogue round 1 | R-1: +2.75 | Helped |
| add random round retaining (R3) during training (ablation) | DiaNA · ChatPedes test, complete dialogue | mAP: +0.70 | Helped |
Surprising findings
- Over one-quarter of LLM-generated dialogue rounds were judged unmatched by Qwen 2.5, motivating cleaning.
- Baselines saturate after a few dialogue rounds while DiaNA keeps improving.
Limitations
- Dialogues are synthetic, generated by LLMs from existing captions rather than real user interactions.
- Baselines were reproduced by concatenating dialogue turns into a single long text, which may disadvantage them.
Experimental setup
Datasets: CUHK-PEDES ChatPedes
Benchmarks: ChatPedes
Metrics: Rank-1 Rank-5 Rank-10 mAP
Baselines: IRRA TBPS-CLIP RDE APTM AUL
Builds on: CUHK-PEDES IRRA CLIP Swin Transformer Llama 3
Closest papers by meaning
- Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval (AAAI 2024 · similarity 0.87)
- Human-centered Interactive Learning via MLLMs for Text-to-Image Person Re-identification (CVPR 2025 · similarity 0.87)
- An Empirical Study of CLIP for Text-Based Person Search (AAAI 2024 · similarity 0.87)
- LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-Identification (ICML 2025 · similarity 0.86)
- Automatic Synthetic Data and Fine-grained Adaptive Feature Alignment for Composed Person Retrieval (NeurIPS 2025 · similarity 0.86)
- UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.