Research map / Vision-language and multimodal / Cross-modal learning
Multi-Modal Disordered Representation Learning Network for Description-Based Person Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MDRL, a description-based person search network with a ViT visual branch and BERT-based textual branch, plus disordered (shuffled) local token grouping and a cross-modality interaction transformer, trained with ID and CMPM losses.
Problem
Description-based person search (retrieving person images from text descriptions) needs discriminative visual and textual representations, but hard part-based splits and external part/pose/attribute models ignore global relations between local parts and make training unstable.
Approach
Shuffle visual and textual token sequences randomly into groups, each concatenated with the global feature, so local features are learned with global context and no extra models; add a cross-modality interaction transformer over concatenated tokens to couple the two branches.
Key results
- Achieves 74.56% Rank-1 on CUHK-PEDES, +1.18 over prior best IRRA (73.38%).
- Achieves 65.88% Rank-1 on ICFG-PEDES, +2.42 over IRRA (63.46%).
- Ablation: DLL(V&T) adds 3.71 Rank-1 points over the global baseline on CUHK-PEDES (70.19 to 73.90).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MDRL instead of IRRA | Description-based person search · CUHK-PEDES | Rank-1 accuracy: 1.18% | Helped |
| Use MDRL instead of IRRA | Description-based person search · ICFG-PEDES | Rank-1 accuracy: 2.42% | Helped |
| Add the disorder local learning module to the global architecture (ablation) | MDRL · CUHK-PEDES | Rank-1 accuracy: +4.37% | Helped |
| Add the visual and textual disorder local learning module (ablation) | MDRL global architecture · CUHK-PEDES | Rank-1 accuracy: 70.19 -> 73.90 | Helped |
| Add the Cross-modality Interaction Module (ablation) | MDRL global architecture · CUHK-PEDES | Rank-1 accuracy: 70.19 -> 71.96 | Helped |
| Vary the number of learnable Transformer encoders from L=1 to L=6 (ablation) | MDRL textual branch · CUHK-PEDES | Rank-1 accuracy: 68.45; 72.06; 74.27; 74.56; 73.26; 74.07 | Mixed |
| Vary disordered visual groups from P=2 to P=8 (ablation) | MDRL visual local learning · CUHK-PEDES | Rank-1 accuracy: 73.26; 73.75; 73.23; 74.56; 73.68 | Mixed |
| Replace disordered visual grouping with horizontal splitting (ablation) | MDRL visual local learning · CUHK-PEDES | Rank-1 accuracy: 73.59 vs 74.56 | Hurt |
| Vary disordered textual groups from Q=2 to Q=8 (ablation) | MDRL textual local learning · CUHK-PEDES | Rank-1 accuracy: 73.23; 74.56; 73.49 | Mixed |
| Replace disordered textual grouping with equidistant splitting (ablation) | MDRL textual local learning · CUHK-PEDES | Rank-1 accuracy: 74.56 vs 73.20 | Hurt |
| Vary the number of Cross-modality Interaction Transformer Encoders from K=1 to K=5 (ablation) | MDRL Cross-modality Interaction Module · CUHK-PEDES | Rank-1 accuracy: 73.78; 74.56; 73.88; 72.90; 73.39 | Mixed |
Surprising findings
- Performance declines when the number of LTE layers exceeds 4 and when CIM layers exceed 2, attributed to overfitting on limited training data.
Limitations
- Gains from disordered grouping depend on the number of groups; too many groups reduce performance.
- Evaluated only on two datasets (CUHK-PEDES, ICFG-PEDES).
Experimental setup
Datasets: CUHK-PEDES ICFG-PEDES
Benchmarks: CUHK-PEDES ICFG-PEDES
Metrics: Rank-1 Rank-5 Rank-10
Baselines: IRRA LGUR AXM-Net ViTAA NAFS+LapsCore MGEL DSSL TIMAM
Builds on: Vision Transformer BERT CMPM loss ID loss
Closest papers by meaning
- Causality-Inspired Invariant Representation Learning for Text-Based Person Retrieval (AAAI 2024 · similarity 0.86)
- Graph-Based Cross-Domain Knowledge Distillation for Cross-Dataset Text-to-Image Person Retrieval (AAAI 2025 · similarity 0.84)
- An Empirical Study of CLIP for Text-Based Person Search (AAAI 2024 · similarity 0.84)
- Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval (AAAI 2024 · similarity 0.84)
- Empowering Visible-Infrared Person Re-Identification with Large Foundation Models (NeurIPS 2024 · similarity 0.83)
- PLIP: Language-Image Pre-training for Person Representation Learning (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.