Research map / Vision-language and multimodal / Cross-modal learning
DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeMo, a multi-modal object ReID framework that extracts multi-granularity features (PIFE), decouples RGB/NIR/TIR features hierarchically via cross-attention (HDM), and fuses them with an attention-triggered mixture of experts (ATMoE).
Problem
Multi-modal object re-identification suffers from dynamic, instance-specific quality changes across RGB, NIR and TIR modalities, and directly weighting or fusing multi-modal features blurs modality-specific information.
Approach
Decouple multi-modal features into unimodal-specific, bimodal-shared and trimodal-shared components using learnable queries with cross-attention, then weight these decoupled features as MoE experts using multi-head attention between a query derived from the decoupled features and their keys rather than a simple gate.
Key results
- On RGBNT201, DeMo (CLIP-based) reaches 79.0% mAP and 82.3% Rank-1, versus 72.3% mAP for the ViT-based TOP-ReID.
- On RGBNT100, DeMo (CLIP-based) reaches 86.2% mAP, outperforming EDITOR's 82.1%.
- On MSVR310, DeMo (CLIP-based) reaches 49.2% mAP, versus 39.5% for RSCNet; in missing-modality settings on RGBNT201 it averages 50.7% mAP, 6.3 points above TOP-ReID.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeMo with a CLIP backbone instead of TOP-ReID with a CLIP backbone | RGBNT201 · RGBNT201 | mAP: 73.3 -> 79.0 | Helped |
| Use DeMo with a ViT backbone instead of TOP-ReID | RGBNT201 · RGBNT201 | mAP: +1.4% | Helped |
| Use DeMo with a CLIP backbone instead of EDITOR | RGBNT100 · RGBNT100 | mAP: 4.1% | Helped |
| Use DeMo with a CLIP backbone instead of TOP-ReID | RGBNT201 missing-modality evaluation · RGBNT201 missing-modality average | mAP: 6.3% | Helped |
| Use DeMo with a CLIP backbone instead of TOP-ReID | RGBNT100 missing-modality evaluation · RGBNT100 missing-modality average | mAP: 9.3% | Helped |
| Add PIFE to the baseline (ablation) | DeMo on RGBNT201 · RGBNT201 | mAP: 70.7 -> 73.0 | Helped |
| Add HDM (ablation) | DeMo on RGBNT201 · RGBNT201 | mAP: 73.0 -> 74.4 | Helped |
| Add ATMoE (ablation) | DeMo on RGBNT201 · RGBNT201 | mAP: 74.4 -> 76.8 | Helped |
| Add the PIFE features to the ATMoE feature (ablation) | DeMo on RGBNT201 · RGBNT201 | mAP: 76.8 -> 79.0 | Helped |
| Use GeM pooling instead of average pooling (ablation) | PIFE on RGBNT201 · RGBNT201 | mAP: 79.0 -> 79.1 | Helped |
| Use max pooling instead of average pooling (ablation) | PIFE on RGBNT201 · RGBNT201 | mAP: 79.0 -> 76.6 | Hurt |
| Add PIFE (ablation) | DeMo on RGBNT201 · RGBNT201 | mAP: 77.7 -> 79.0 | Helped |
Surprising findings
- A simple concatenation-based gate outperforms the addition-based variant, and attention gating with 1 head already beats the simple gate.
- Increasing the number of heads beyond 4 does not consistently help on RGBNT201 (8 heads gives 78.2% mAP vs 79.0% at 4 heads).
Limitations
- Gains over ViT-based baselines are partly attributed to CLIP backbone; the ViT-based DeMo result (73.7% mAP on RGBNT201) is a smaller margin.
- Number of heads and expert configurations were tuned on the test benchmarks, with best settings differing across datasets.
- Evaluated only on three benchmarks, all with RGB/NIR/TIR modalities.
Experimental setup
Datasets: RGBNT201 RGBNT100 MSVR310
Benchmarks: RGBNT201 RGBNT100 MSVR310
Metrics: mAP CMC Rank-1 Rank-5 Rank-10
Baselines: TOP-ReID EDITOR RSCNet HTT LRMM UniCat PFNet HAMNet DENet IEEE TransReID CCNet
Builds on: Mixture of Experts CLIP ViT TOP-ReID
Closest papers by meaning
- MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification (NeurIPS 2025 · similarity 0.90)
- Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification (CVPR 2024 · similarity 0.90)
- Multi-Modal Object Re-identification via Sparse Mixture-of-Experts (ICML 2025 · similarity 0.89)
- MambaPro: Multi-Modal Object Re-identification with Mamba Aggregation and Synergistic Prompt (AAAI 2025 · similarity 0.88)
- UGG-ReID: Uncertainty-Guided Graph Model for Multi-Modal Object Re-Identification (NeurIPS 2025 · similarity 0.88)
- IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identification (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.