Research map / Detection and segmentation / Tracking
Foundation Model Driven Appearance Extraction for Robust Multiple Object Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FDTracker, a tracking-by-detection MOT method whose appearance extractor is a two-stage trained temporal module on frozen CLIP that uses attribute instructions and then multi-frame ReID, fused with IOU distance and lost-tracklet reconnection.
Problem
Multiple object tracking struggles with occlusion and blur in complex scenes; existing appearance features are weak and ignore long-term temporal information.
Approach
Use a frozen multimodal foundation model (CLIP) with language instructions and a temporal appearance extraction module, trained in two stages (single-image attribute perception on SYNTH-PEDES, then multi-frame contrastive ReID on MOT17/DanceTrack), and fuse the resulting appearance distance with IOU distance for association and reconnection of lost tracklets.
Key results
- On MOT17 private test, FDTracker reaches 81.8 MOTA, 79.2 IDF1, 64.2 HOTA, improving over its ByteTrack baseline (80.3 MOTA, 63.1 HOTA).
- On DanceTrack test, FDTracker achieves 92.1 MOTA and 70.6 HOTA, best among compared methods.
- Ablation: two-stage TAE plus tracklet management raises MOTA from 87.0 to 90.2 and cuts ID switches from 477 to 178 over the ByteTrack baseline on the validation split.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the two-stage TAE and tracklet management to ByteTrack | ByteTrack with YOLOX-m · MOT17 validation | IDF1: 80.1 -> 86.1 | Helped |
| Add the two-stage TAE and tracklet management to ByteTrack | ByteTrack with YOLOX-m · MOT17 validation | MOTA: 87.0 -> 90.2 | Helped |
| Use only stage 1 TAE training instead of both stages (ablation) | FDTracker TAE · MOT17 validation | MOTA: 90.2 -> 77.6 | Hurt |
| Use only stage 2 TAE training instead of both stages (ablation) | FDTracker TAE · MOT17 validation | MOTA: 90.2 -> 89.6 | Hurt |
| Remove stage 1 TAE training from the two-stage training (ablation) | FDTracker TAE · MOT17 validation | IDF1: 86.1 -> 81.6 | Hurt |
| Remove tracklet management (ablation) | FDTracker · MOT17 validation | IDF1: 83.4 -> 86.1 | Hurt |
| Use BLIP2 instead of CLIP as the foundation model (ablation) | FDTracker appearance feature extractor · MOT17 validation | MOTA: 90.2 -> 90.4 | No clear effect |
| Use BLIP instead of CLIP as the foundation model (ablation) | FDTracker appearance feature extractor · MOT17 validation | IDF1: 86.1 -> 86.4 | No clear effect |
| Use sentence outputs instead of word outputs in stage 1 (ablation) | FDTracker TAE · MOT17 validation | MOTA: 90.2 -> 90.3 | No clear effect |
| Use sentence outputs instead of word outputs in stage 1 (ablation) | FDTracker TAE · MOT17 validation | IDF1: 86.1 -> 85.5 | Hurt |
| Add FDTracker appearance extraction and association to ByteTrack | ByteTrack · MOT17 test | HOTA: 63.1 -> 64.2 | Helped |
| Add FDTracker appearance extraction and association to ByteTrack | ByteTrack · DanceTrack test | HOTA: 47.7 -> 70.6 | Helped |
Surprising findings
- Stage-1 attribute training alone degrades performance (MOTA 77.6 vs 87.0 baseline), attributed to domain gap between ReID and MOT data.
- Larger BLIP2 (4.1B) gives only marginal gain over CLIP (90.4 vs 90.2 MOTA), suggesting little benefit from scale for this task.
Limitations
- Attribute descriptions are limited to six manually designed pedestrian attributes, lacking linguistic diversity.
- Reconnection of lost tracklets can cause incorrect ID switches in some cases.
- Evaluated mainly on pedestrian benchmarks; generalization to other classes is only shown on BDD100k with a modified second stage.
Experimental setup
Datasets: SYNTH-PEDES MOT17 DanceTrack MOT20 BDD100k
Benchmarks: MOT17 MOT20 DanceTrack BDD100k
Metrics: MOTA IDF1 HOTA FP FN ID.Sw mMOTA
Baselines: ByteTrack BoT-SORT UTM MOTRv2 MOTR GHOST OC-SORT DNMOT MeMOT FairMOT
Builds on: ByteTrack CLIP YOLOX Deep SORT
Closest papers by meaning
- OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer (ICLR 2025 · similarity 0.87)
- OVT-B: A New Large-Scale Benchmark for Open-Vocabulary Multi-Object Tracking (NeurIPS 2024 · similarity 0.85)
- Robust Tracking via Mamba-based Context-aware Token Learning (AAAI 2025 · similarity 0.85)
- Multiple Object Tracking as ID Prediction (CVPR 2025 · similarity 0.85)
- SMILEtrack: SiMIlarity LEarning for Occlusion-Aware Multiple Object Tracking (AAAI 2024 · similarity 0.85)
- Unifying Visual and Vision-Language Tracking via Contrastive Learning (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.