Research map / Vision-language and multimodal / Cross-modal learning
ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ORBench, a five-modality person re-identification dataset (RGB, infrared, color pencil, sketch, text), and ReID5o, a single CLIP-based model with modality-specific low-rank experts and feature mixture that retrieves with arbitrary query modality combinations.
Problem
Person re-identification is typically limited to one or two query modalities, and no dataset or method supports retrieval from arbitrary combinations of multiple query modalities (omni multi-modal ReID).
Approach
A multi-modal tokenizing assembler feeds modality-specific tokenizers into a shared CLIP encoder augmented with per-modality low-rank experts (activated by a modality control signal), followed by a transformer-based feature mixture over modality combinations, trained with SDM plus identity classification losses across all query combinations aligned to RGB.
Key results
- ReID5o reaches 58.09 mAP in single-modal search and 86.35 mAP in quad-modal search on ORBench, outperforming baselines such as UNIReID and IRRA in all four modes.
- Adding modalities to a text query raises mAP, e.g. T alone 54.88 mAP versus T+I+C+S 87.46 mAP.
- Multi-expert router with rank r=4 adds 2.36M parameters and improves MM-1 mAP from 52.80 to 58.09 over the no-router variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ReID5o's unified encoding, multi-expert routing, feature mixture, and learning strategy | ORBench person Re-identification · ORBench MM-1 | mAP: 51.42 -> 58.09 | Helped |
| add ReID5o's unified encoding, multi-expert routing, feature mixture, and learning strategy | ORBench person Re-identification · ORBench MM-2 | mAP: 68.83 -> 75.26 | Helped |
| add ReID5o's unified encoding, multi-expert routing, feature mixture, and learning strategy | ORBench person Re-identification · ORBench MM-3 | mAP: 75.86 -> 82.83 | Helped |
| add ReID5o's unified encoding, multi-expert routing, feature mixture, and learning strategy | ORBench person Re-identification · ORBench MM-4 | mAP: 79.15 -> 86.35 | Helped |
| add multi-modal tokenizing assembler (ablation) | ReID5o · ORBench MM-1 | mAP: 49.24 -> 51.85 | Helped |
| add multi-expert router to MTA (ablation) | ReID5o · ORBench MM-1 | mAP: 51.85 -> 56.99 | Helped |
| add feature mixture to MTA (ablation) | ReID5o · ORBench MM-1 | mAP: 51.85 -> 52.80 | Helped |
| vary multi-expert router rank from 4 to 8 (ablation) | ReID5o · ORBench MM-1 | mAP: 58.09 -> 57.59 | Hurt |
| vary multi-expert router rank from 4 to 16 (ablation) | ReID5o · ORBench MM-1 | mAP: 58.09 -> 57.81 | Hurt |
| vary multi-expert router rank from 4 to 32 (ablation) | ReID5o · ORBench MM-1 | mAP: 58.09 -> 57.56 | Hurt |
| replace multi-expert router with separate encoders for each modality (ablation) | ReID5o · ORBench MM-1 | mAP: 58.09 -> 52.42 | Hurt |
| replace SDM+IC with ITC (ablation) | ReID5o · ORBench MM-1 | mAP: 58.09 -> 53.44 | Hurt |
Surprising findings
- Color pencil queries combined with text give the largest gain among added modalities, and infrared complements color pencil better than sketch does.
Limitations
- Dataset built under idealized assumptions (viewed sketches rather than forensic sketches), so noisy real-world queries are not captured.
- Limited to five predefined modalities; no incremental learning of new modalities.
- Closed-set identification only; open-set unseen identities not addressed.
Experimental setup
Datasets: ORBench SYSU-MM01 LLCM CUHK-PEDES ICFG-PEDES UFine6926 PKU-Sketch Tri-CUHK-PEDES AIO
Benchmarks: ORBench CUHK-PEDES ICFG-PEDES UFine6926 PKU-Sketch
Metrics: CMC rank-1 rank-5 rank-10 mAP mINP
Baselines: CLIP PLIP IRRA RDE Meta-TF ImageBind UNIReID AIO
Builds on: CLIP SDM loss LoRA
Closest papers by meaning
- Multi-Prompts Learning with Cross-Modal Alignment for Attribute-Based Person Re-identification (AAAI 2024 · similarity 0.89)
- Heterogeneous Test-Time Training for Multi-Modal Person Re-identification (AAAI 2024 · similarity 0.89)
- Empowering Visible-Infrared Person Re-Identification with Large Foundation Models (NeurIPS 2024 · similarity 0.88)
- Modeling Thousands of Human Annotators for Generalizable Text-to-Image Person Re-identification (CVPR 2025 · similarity 0.88)
- LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-Identification (ICML 2025 · similarity 0.87)
- FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.