Research map / Detection and segmentation / Tracking
iKUN: Speak to Trackers without Retraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes iKUN, a plug-and-play referring module placed after frozen off-the-shelf trackers, plus a neural Kalman filter and test-time similarity calibration, and a new Refer-Dance dataset for referring multi-object tracking.
Problem
Referring multi-object tracking (RMOT) requires tracking objects matching text descriptions, but prior methods integrate a textual module into the tracker and must retrain the whole framework, causing task competition, engineering cost, and training cost; descriptions also follow an open-set long-ta
Approach
Decouple tracking from referring: freeze an off-the-shelf tracker and train a two-stream CLIP-style scorer whose visual features are modulated by text (knowledge unification module), with a neural Kalman filter that learns process/observation noise and a test-time pseudo-frequency similarity calibration.
Key results
- On Refer-KITTI, NeuralSORT+iKUN reaches 44.56% HOTA with YOLOv8 detector versus 38.06% for TransRMOT, and 48.84% HOTA with DeformableDETR detector.
- On Refer-Dance, the method reaches 29.06% HOTA versus 9.58% for TransRMOT.
- Neural Kalman filter raises ByteTrack HOTA on KITTI from 38.87% to 44.89% and on DanceTrack from 46.88% to 54.52%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Insert iKUN after the tracker and apply similarity calibration | NeuralSORT · Refer-KITTI | HOTA: 38.06 -> 44.56 | Helped |
| Apply iKUN with ByteTrack and NKF | ByteTrack · Refer-Dance | HOTA: 9.58% -> 29.06% | Helped |
| Replace vanilla Kalman filter with neural Kalman filter | ByteTrack · KITTI | HOTA: 38.87 -> 44.89 | Helped |
| Replace vanilla Kalman filter with neural Kalman filter | ByteTrack · DanceTrack | HOTA: 46.88 -> 54.52 | Helped |
| Use cascade attention KUM instead of visual-only feature unification (ablation) | iKUN · Refer-KITTI | HOTA: 39.09 -> 43.75 | Helped |
| Use cross-correlation KUM instead of visual-only feature unification (ablation) | iKUN · Refer-KITTI | HOTA: 39.09 -> 40.14 | Helped |
| Use text-first modulation KUM instead of visual-only feature unification (ablation) | iKUN · Refer-KITTI | HOTA: 39.09 -> 40.90 | Helped |
| Add NKF (ablation) | DeepSORT · KITTI Car | HOTA: 58.58 -> 59.90 | Helped |
| Add NKF (ablation) | DeepSORT · KITTI Pedestrian | HOTA: 45.03 -> 48.53 | Helped |
| Add extra exiting decision (ablation) | DeepSORT · KITTI Car | HOTA: 58.58 -> 58.69 | No clear effect |
| Add extra velocity cost (ablation) | DeepSORT · KITTI Pedestrian | HOTA: 45.03 -> 47.25 | Helped |
| Add linear interpolation (ablation) | DeepSORT · KITTI Car | HOTA: 58.58 -> 57.64 | Hurt |
Surprising findings
- Text-first modulation gives the best AssA but poor DetA, showing a trade-off among KUM designs.
- Similarity calibration hyper-parameters have little effect on performance, the method is robust to their values.
Limitations
- Multi-stage framework has many nontrivial hyperparameters across tracking, referring and post-processing components.
- Limited temporal modelling capability constrains performance on motion-related queries.
Experimental setup
Datasets: Refer-KITTI KITTI DanceTrack Refer-Dance
Benchmarks: Refer-KITTI Refer-Dance KITTI DanceTrack
Metrics: HOTA DetA AssA MOTA IDF1 DetRe DetPr AssRe AssPr
Baselines: TransRMOT FairMOT ByteTrack CSTrack TransTrack TrackFormer OC-SORT StrongSORT Deep OC-SORT
Builds on: CLIP TransRMOT DeepSORT Kalman filter focal loss
Closest papers by meaning
- Cross-View Referring Multi-Object Tracking (AAAI 2025 · similarity 0.85)
- ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model (NeurIPS 2024 · similarity 0.84)
- HFF-Tracker: A Hierarchical Fine-grained Fusion Tracker for Referring Multi-Object Tracking (AAAI 2025 · similarity 0.84)
- UCMCTrack: Multi-Object Tracking with Uniform Camera Motion Compensation (AAAI 2024 · similarity 0.83)
- OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer (ICLR 2025 · similarity 0.82)
- Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.