Research map / Detection and segmentation / Tracking
Multiple Object Tracking as ID Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates multi-object tracking association as in-context ID prediction, using a Deformable DETR detector plus a learnable ID dictionary and transformer ID decoder trained end-to-end (MOTIP).
Problem
Multi-object tracking association relies on handcrafted heuristics and cost matrices; end-to-end learned association struggles with generalization to unseen trajectory IDs.
Approach
Treat association as in-context ID prediction: learnable ID embeddings are attached to historical trajectory features, and a transformer decoder predicts the ID label of current detections via cross-entropy, so random label assignment at inference keeps predictions within the training label range.
Key results
- On DanceTrack test (without extra data), MOTIP reaches 69.6 HOTA and 60.4 AssA, outperforming CO-MOT's 65.3 HOTA.
- On SportsMOT test (without extra data), MOTIP achieves 72.6 HOTA, above MeMOTR's 68.8.
- On BFT test, MOTIP sets a new state of the art with 70.5 HOTA and 71.8 AssA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MOTIP instead of CO-MOT | DanceTrack test tracking · DanceTrack test | HOTA: 65.3 -> 69.6 | Helped |
| Use MOTIP instead of MeMOTR | SportsMOT test tracking · SportsMOT test | HOTA: 68.8 -> 72.6 | Helped |
| Use MOTIP instead of OC-SORT | BFT test tracking · BFT test | HOTA: 66.8 -> 70.5 | Helped |
| Add self-attention to the ID Decoder (ablation) | MOTIP on DanceTrack · DanceTrack | HOTA: 57.7 -> 59.7 | Helped |
| Add the Hungarian algorithm (ablation) | MOTIP on DanceTrack · DanceTrack | HOTA: 59.7 -> 60.8 | Helped |
| Add trajectory augmentation (ablation) | MOTIP on DanceTrack · DanceTrack | HOTA: 59.5 -> 62.2 | Helped |
| Use in-context ID prediction instead of ReID classification or contrastive learning (ablation) | Two-stage MOT on DanceTrack · DanceTrack | HOTA: 29.4; 41.0 -> 55.4 | Helped |
| Use in-context ID prediction instead of ReID formulations (ablation) | One-stage MOT on DanceTrack · DanceTrack | HOTA: 41.0; 49.8; 51.2; 50.6; 52.6; 54.5 -> 59.5 | Helped |
| Set trajectory random occlusion probability to 0.5 rather than 0.0 (ablation) | MOTIP on DanceTrack · DanceTrack | HOTA: 59.5 -> 60.7 | Helped |
| Set trajectory random occlusion probability to 1.0 rather than 0.5 (ablation) | MOTIP on DanceTrack · DanceTrack | HOTA: 60.7 -> 58.6 | Hurt |
| Set trajectory random switch probability to 0.2 rather than 0.0, with λocc=0.5 (ablation) | MOTIP on DanceTrack · DanceTrack | HOTA: 60.7 -> 61.6 | Helped |
| Set trajectory random switch probability to 0.5 rather than 0.2, with λocc=0.5 (ablation) | MOTIP on DanceTrack · DanceTrack | HOTA: 61.6 -> 62.2 | Helped |
Surprising findings
- Adding the Hungarian algorithm to MOTIP's inference gives little benefit, suggesting the learned ID decoder already resolves association globally.
- Self-attention in the ID decoder is important: it lets current detections exchange identity information and prevents confusion among similar targets.
Limitations
- Fixed ID dictionary size K (50) may be insufficient in crowded scenes.
- Evaluated only with a Deformable DETR ResNet-50 detector and relies on object-level features alone as tracking cues.
- Excludes extra detection datasets for training, so comparisons with joint-training methods are partial.
Experimental setup
Datasets: DanceTrack SportsMOT BFT COCO
Benchmarks: DanceTrack SportsMOT BFT
Metrics: HOTA DetA AssA MOTA IDF1
Baselines: FairMOT ByteTrack OC-SORT MeMOTR CO-MOT MOTRv2 MOTRv3 QDTrack Hybrid-SORT DiffMOT
Builds on: Deformable DETR MOTR MeMOTR Transformer decoder
Closest papers by meaning
- From Detection to Association: Learning Discriminative Object Embeddings for Multi-Object Tracking (CVPR 2026 · similarity 0.90)
- DiffusionTrack: Diffusion Model for Multi-Object Tracking (AAAI 2024 · similarity 0.89)
- OVT-B: A New Large-Scale Benchmark for Open-Vocabulary Multi-Object Tracking (NeurIPS 2024 · similarity 0.89)
- Focusing on Tracks for Online Multi-Object Tracking (CVPR 2025 · similarity 0.88)
- ADA-Track: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and Association (CVPR 2024 · similarity 0.88)
- Self-Supervised Multi-Object Tracking with Path Consistency (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.