Research map / Detection and segmentation / Tracking
MTMMC: A Large-Scale Real-World Multi-Modal Camera Tracking Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MTMMC, a large-scale real-world multi-modal (RGB + thermal) multi-target multi-camera tracking dataset from 16 cameras across campus and factory settings, with baselines for detection, re-ID, MOT, multi-modal learning, and MTMC.
Problem
Existing multi-camera tracking datasets are synthetic, small, low-resolution, or collected in controlled overlapping camera setups, and DukeMTMC was withdrawn, leaving no large real-world MTMC benchmark, especially with multi-modal data.
Approach
A real-world 16-camera RGB+thermal dataset (3M frames, 3,669 identities) recorded in campus and factory environments with varied time, weather, and season, plus new multi-modal learning setups (modality fusion and modality drop) evaluated on it.
Key results
- Training YOLOX on MTMMC-Person improves MOT17 mAP from 34.2 (COCO-Person) to 38.3.
- Adding thermal via feature-level fusion raises RGB-only QDTrack IDF1 on MTMMC from 53.0 to 53.9 and MOTA from 84.5 to 86.0.
- Multi-modal contrastive modality-drop training yields IDF1 59.7 / MOTA 48.4 on MOT17 without fine-tuning, versus 55.3 / 43.6 for RGB-only baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Faster RCNN on MTMMC-Person instead of COCO-Person | Faster RCNN · MOT17 | mAP: 29.8 -> 31.3 | Helped |
| Train YOLOX on MTMMC-Person instead of COCO-Person | YOLOX · MOT17 | mAP: 34.2 -> 38.3 | Helped |
| Train AGW on MTMMC-reID instead of MSMT17 | AGW · Market-1501 | Rank 1: 64.3 -> 66.5 | Helped |
| Pre-train QDTrack on MTMMC instead of MOTSynth | QDTrack · MOT17, without fine-tuning | MOTA: 54.1 -> 55.3 | Helped |
| Combine MTMMC and MOTSynth for pre-training | QDTrack · MOT17, with fine-tuning | IDF1: 70.8 -> 72.0 | Helped |
| Fuse RGB and thermal at input level | QDTrack · MTMMC | IDF1: 53.0 -> 54.0 | Helped |
| Fuse RGB and thermal at feature level | QDTrack · MTMMC | MOTA: 84.5 -> 86.0 | Helped |
| Train with knowledge distillation from RGB and thermal | QDTrack · MTMMC → MOT17, without fine-tuning | IDF1: 55.3 -> 55.1 | Hurt |
| Train with multi-modal reconstruction | QDTrack · MTMMC → MOT17, without fine-tuning | IDF1: 55.3 -> 57.9 | Helped |
| Train with multi-modal contrastive learning | QDTrack · MTMMC → MOT17, without fine-tuning | IDF1: 55.3 -> 59.7 | Helped |
| Fine-tune the knowledge-distilled model on RGB data | QDTrack · MTMMC → MOT17 | IDF1: 55.1 -> 70.5 | Helped |
| Add RGB-thermal fusion to hierarchical clustering | H. Cluster · MTMMC | IDF1: 41.6 -> 43.5 | Helped |
Surprising findings
- Models trained on MTMMC generalize better to MOT17 than models trained on MOT17 itself, e.g., ByteTrack IDF1 69.1 vs 40.2 on MTMMC.
Limitations
- Multi-modal gains in MTMC are modest (e.g., IDF1 43.7 to 43.5 range) and baselines are preliminary.
- Modality-drop fine-tuning results are not consistently better than RGB-only baseline without fine-tuning.
Experimental setup
Datasets: MTMMC MOT17 COCO-Person CrowdHuman MOTSynth MSMT17 Market-1501
Benchmarks: MTMMC MOT17 Market-1501 MSMT17
Metrics: mAP Rank-1 IDF1 MOTA FP FN IDs
Baselines: JDE QDTrack CenterTrack ByteTrack Faster RCNN YOLOX AGW hierarchical clustering multi-camera association optimization-based multi-camera association
Builds on: QDTrack DeepSORT BoT AGW DukeMTMC
Closest papers by meaning
- All-Day Multi-Camera Multi-Target Tracking (CVPR 2025 · similarity 0.85)
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.84)
- Leader360V: A Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment (NeurIPS 2025 · similarity 0.84)
- Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm (NeurIPS 2025 · similarity 0.83)
- DiffusionTrack: Diffusion Model for Multi-Object Tracking (AAAI 2024 · similarity 0.83)
- UCMCTrack: Multi-Object Tracking with Uniform Camera Motion Compensation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.