Research map / Vision-language and multimodal / Cross-modal learning
Multi-Modal Object Re-identification via Sparse Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MFRNet, a CLIP-ViT-based multi-modal object re-ID network that uses sparse mixture-of-experts for pixel-level cross-modal feature generation (FFM) and modality-shared/specific representation (FRM), evaluated on RGBNT201, RGBNT100, and MSVR310.
Problem
Multi-modal object re-identification suffers from insufficient pixel-level cross-modal interaction (prior work fuses high-level semantics) and difficulty balancing modality-shared and modality-specific features in a unified network.
Approach
Applies sparse mixture-of-experts at two places: a mixture of simple convolutional feature generators that reconstructs missing/other-modality tokens before the ViT (FFM), and a MoE-enhanced RepAdapter in the final ViT layer that routes tokens to representation experts (FRM), keeping parameter overhead low.
Key results
- On RGBNT201, MFRNet reaches 80.7% mAP and 83.5% R-1, improving over TOP-ReID by 8.4 mAP and 6.9 R-1 points.
- On MSVR310, mAP rises from 39.5 (RSCNet) to 50.6, an 11.1-point gain; on RGBNT100 mAP reaches 88.2 vs 82.3 for RSCNet.
- With 57.1M params and 22.1 GFLOPs, MFRNet is cheaper than TOP-ReID (278.2M params) and EDITOR (117.5M params).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MFRNet instead of TOP-ReID | RGBNT201 object ReID · RGBNT201 | mAP: 72.3 -> 80.7 | Helped |
| Use MFRNet instead of TOP-ReID | RGBNT201 object ReID · RGBNT201 | R-1: 76.6 -> 83.6 | Helped |
| Use MFRNet instead of RSCNet | RGBNT100 vehicle ReID · RGBNT100 | mAP: 82.3 -> 88.2 | Helped |
| Use MFRNet instead of RSCNet | RGBNT100 vehicle ReID · RGBNT100 | R-1: 96.6 -> 97.4 | Helped |
| Use MFRNet instead of RSCNet | MSVR310 vehicle ReID · MSVR310 | mAP: 39.5 -> 50.6 | Helped |
| Use MFRNet instead of CCNet | MSVR310 vehicle ReID · MSVR310 | R-1: 55.2 -> 64.8 | Helped |
| Add FFM (ablation) | Shared ViT baseline on RGBNT201 · RGBNT201 | mAP: 69.2 -> 75.1 | Helped |
| Add FRM (ablation) | Shared ViT baseline on RGBNT201 · RGBNT201 | mAP: 69.2 -> 77.8 | Helped |
| Add FFM and FRM together (ablation) | Shared ViT baseline on RGBNT201 · RGBNT201 | mAP: 69.2 -> 80.7 | Helped |
| Use 6 rather than 3 FRM experts (ablation) | FRM on RGBNT201 · RGBNT201 | mAP: 77.6 -> 77.8 | Helped |
| Increase FRM experts from 6 to 9 (ablation) | FRM on RGBNT201 · RGBNT201 | mAP: 77.8 -> 75.1 | Hurt |
| Place FRM only at layer 12 rather than layers 10, 11, and 12 (ablation) | FRM on RGBNT201 · RGBNT201 | mAP: 71.6 -> 77.8 | Helped |
Surprising findings
- Placing the feature fusion module before the ViT (pixel level) works far better than after ViT (post-fusion 83.8 avg vs 87.5 avg), and inserting it at layer 3 drops performance sharply (58.7 avg).
Limitations
- FFM adds 3.6M parameters and 1.4 GFLOPs, which the authors note still need reduction.
- Performance is sensitive to hyperparameters such as the number of experts, lambda, and module placement; gains on RGBNT201 are reported as single-run results without variance.
Experimental setup
Datasets: RGBNT201 RGBNT100 MSVR310
Benchmarks: RGBNT201 RGBNT100 MSVR310
Metrics: mAP CMC Rank-1 Rank-5 Rank-10 parameters FLOPs
Baselines: TOP-ReID EDITOR RSCNet HTT PFNet IEEE DENet UniCat TransReID CCNet
Builds on: CLIP RepAdapter Tutel MoE TOP-ReID EDITOR
Closest papers by meaning
- MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification (NeurIPS 2025 · similarity 0.91)
- DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification (AAAI 2025 · similarity 0.89)
- Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification (CVPR 2024 · similarity 0.89)
- MambaPro: Multi-Modal Object Re-identification with Mamba Aggregation and Synergistic Prompt (AAAI 2025 · similarity 0.88)
- FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification (ICML 2025 · similarity 0.87)
- IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identification (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.