Research map / Detection and segmentation / Tracking
HFF-Tracker: A Hierarchical Fine-grained Fusion Tracker for Referring Multi-Object Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HFF-Tracker, an end-to-end referring multi-object tracker with hierarchical pixel/word/sentence-level visual-text fusion, text-guided decoder and prediction head, and a Look-Back training strategy, reaching SOTA HOTA on Refer-KITTI and Refer-KITTI-V2.
Problem
Referring multi-object tracking requires tracking all objects matching a language expression, but existing trackers fuse text and image features shallowly and underuse the text and the labeled training data.
Approach
Fine-grained fusion of pixel-level visual features with word-level and sentence-level text features at multiple backbone levels (HMMF), injecting text semantics into decoder queries (TGD) and the referring prediction branch (TGPH), plus an adaptive Look-Back training scheme that retrains on high-loss samples and unused clip-expression pairs.
Key results
- On Refer-KITTI, HFF-Tracker reaches 52.41 HOTA, 41.29 DetA and 66.65 AssA, versus 50.91 HOTA for TempRMOT (+2.16 HOTA).
- On Refer-KITTI-V2, HFF-Tracker reaches 36.18 HOTA and 24.64 DetA, surpassing TempRMOT (34.72 HOTA, 22.52 DetA).
- Ablation: adding HMMF, TGD+TGPH, Temporal Enhancement and Look-Back incrementally raises HOTA from 46.56 (TransRMOT) to 52.41.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add HMMF (ablation) | TransRMOT · Refer-KITTI | HOTA: 46.56 -> 47.53 | Helped |
| Add TGD to the model with HMMF (ablation) | TransRMOT · Refer-KITTI | HOTA: 47.53 -> 47.99 | Helped |
| Add TGPH to the model with HMMF and TGD (ablation) | TransRMOT · Refer-KITTI | HOTA: 47.99 -> 48.62 | Helped |
| Add the Temporal Enhancement module to the model with HMMF, TGD, and TGPH (ablation) | TransRMOT · Refer-KITTI | HOTA: 48.62 -> 51.62 | Helped |
| Add Look-Back training to the model with HMMF, TGD, TGPH, and Temp. (ablation) | TransRMOT · Refer-KITTI | HOTA: 51.62 -> 52.41 | Helped |
| Add Look-Back-Hard (ablation) | The model with Temporal Enhancement · Refer-KITTI | HOTA: 51.62 -> 52.07 | Helped |
| Add Look-Back-Remain to the model with Look-Back-Hard (ablation) | The model with Temporal Enhancement · Refer-KITTI | HOTA: 52.07 -> 52.41 | Helped |
| Add the full HFF-Tracker design | RMOT tracking methods · Refer-KITTI-V2 | HOTA: 34.72 -> 36.18 | Helped |
Surprising findings
- Look-Back-Remain slightly lowers AssA (by 0.35) even though it improves HOTA and DetA.
- iKUN scores only 10.32 HOTA on Refer-KITTI-V2 despite 38.06 on Refer-KITTI, indicating dependence on its off-the-shelf tracker.
Limitations
- Gains over TempRMOT on Refer-KITTI-V2 are mixed, with AssA lower than TransRMOT and TempRMOT.
- Evaluated only on KITTI-based RMOT benchmarks, not on other RMOT datasets.
Experimental setup
Datasets: Refer-KITTI Refer-KITTI-V2
Benchmarks: Refer-KITTI Refer-KITTI-V2
Metrics: HOTA DetA AssA DetRe DetPr AssRe AssPr LocA
Baselines: DeepSORT FairMOT TransTrack ByteTrack CStrack TrackFormer TransRMOT iKUN EchoTrack DeepRMOT MLS-Track TempRMOT
Builds on: TransRMOT MOTR Deformable DETR TempRMOT RoBERTa
Closest papers by meaning
- Cross-View Referring Multi-Object Tracking (AAAI 2025 · similarity 0.85)
- ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model (NeurIPS 2024 · similarity 0.85)
- OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer (ICLR 2025 · similarity 0.85)
- MITracker: Multi-View Integration for Visual Object Tracking (CVPR 2025 · similarity 0.84)
- Foundation Model Driven Appearance Extraction for Robust Multiple Object Tracking (AAAI 2025 · similarity 0.84)
- Unifying Visual and Vision-Language Tracking via Contrastive Learning (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.