Research map / Vision-language and multimodal / Cross-modal learning
MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MDReID, a ViT-based any-to-any multi-spectral object ReID framework that decouples modality-shared and modality-specific tokens, trained with orthogonality and knowledge discrepancy losses, to handle both modality-matched and mismatched queries and galleries.
Problem
Multi-spectral object re-identification assumes query and gallery share the same modalities (RGB, NIR, TIR); real deployments have modality-mismatched query-gallery pairs, which existing methods handle poorly.
Approach
Prepend modality-specific and modality-shared learnable tokens per modality to a CLIP ViT, represent each sample as a zero-padded fixed-length feature vector with an availability mask, and add a representation orthogonality loss (ROL) and knowledge discrepancy loss (KDL) so shared and specific features are orthogonal yet complementary.
Key results
- MDReID reaches 82.1 mAP on RGBNT201 (vs. 72.3 for TOP-ReID) and 85.3 mAP on RGBNT100 in modality-matched retrieval.
- In modality-mismatched scenarios, average mAP improves over TOP-ReID by 3.4% on RGBNT201, 11.8% on RGBNT100, and 10.9% on MSVR310.
- Ablation on RGBNT201 shows MDL alone raises average mAP from 27.8 to 39.4, and adding both ROL and KDL reaches 43.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MDReID instead of TOP-ReID | RGBNT201 modality-mismatched average · RGBNT201, four modality-mismatched scenarios | mAP: 18.2 -> 21.6 | Helped |
| Use MDReID instead of TOP-ReID | RGBNT100 modality-mismatched average · RGBNT100, four modality-mismatched scenarios | mAP: 26.8 -> 38.6 | Helped |
| Use MDReID instead of TOP-ReID | MSVR310 modality-mismatched average · MSVR310, four modality-mismatched scenarios | mAP: 11.2 -> 22.1 | Helped |
| Add modality-decoupled learning (MDL) to the shared ViT baseline (ablation) | MDReID · RGBNT201, 8 evaluation settings | mAP: +11.5% | Helped |
| Use separate modality-specific classifiers instead of one classifier for all modalities (ablation) | MDReID · RGBNT201, 8 evaluation settings | mAP: +13.4% | Helped |
| Add ROL to MDL without KDL (ablation) | MDReID · RGBNT201, 8 evaluation settings | mAP: 39.4 -> 41.2 | Helped |
| Add KDL to MDL without ROL (ablation) | MDReID · RGBNT201, 8 evaluation settings | mAP: 39.4 -> 39.9 | Helped |
| Combine ROL and KDL with MDL (ablation) | MDReID · RGBNT201, 8 evaluation settings | mAP: 39.4 -> 43.2 | Helped |
| Set ROL weight w1 to 1.5 rather than 0.5, 1.0, 2.0, or 3.0 (ablation) | MDReID · RGBNT201, 8 evaluation settings | mAP: 39.6; 40.3; 41.2; 38.9; 38.2 | Mixed |
| Set KDL weight w2 to 5.25 rather than 4.5, 5.0, 5.5, or 6.0 (ablation) | MDReID · RGBNT201, 8 evaluation settings | mAP: 41.7; 40.2; 43.2; 41.6; 41.9 | Mixed |
| Combine shared and modality-specific features instead of using either alone (ablation) | MDReID · RGBNT201, RT-to-NT | mAP: 52.0; 52.7 -> 53.1 | Helped |
Surprising findings
- EDITOR, designed for modality-matched retrieval, performs very poorly in mismatched settings (e.g., 2.8 mAP on RT-to-N for RGBNT201).
Limitations
- Current accuracy is insufficient for real-world deployment according to the authors.
- Existing datasets are limited in scale and modality diversity, so generalization is unclear.
- Only three modalities (RGB, NIR, TIR) and three datasets are evaluated.
Experimental setup
Datasets: RGBNT201 RGBNT100 MSVR310
Benchmarks: RGBNT201 RGBNT100 MSVR310
Metrics: mAP R-1 R-5 R-10
Baselines: TOP-ReID EDITOR HTT RSCNet CCNet DEEN UniCat PHT
Builds on: TOP-ReID CLIP Triplet loss Label smoothing cross-entropy
Closest papers by meaning
- Multi-Modal Object Re-identification via Sparse Mixture-of-Experts (ICML 2025 · similarity 0.91)
- DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification (AAAI 2025 · similarity 0.90)
- Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification (CVPR 2024 · similarity 0.89)
- IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identification (CVPR 2025 · similarity 0.87)
- Implicit Discriminative Knowledge Learning for Visible-Infrared Person Re-Identification (CVPR 2024 · similarity 0.87)
- Empowering Visible-Infrared Person Re-Identification with Large Foundation Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.