Research map / Detection and segmentation / Tracking
OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OVTR, the first end-to-end open-vocabulary multi-object tracker built on MOTR, propagating category information across frames and distilling CLIP image embeddings into a dual-branch decoder.
Problem
Open-vocabulary multiple object tracking suffers from unstable classification across frames, reliance on tracking-by-detection pipelines with post-processing, and costly preprocessing to extract CLIP embeddings for novel categories.
Approach
An iterative query-based tracker (MOTR-style) propagates category information through track queries (category information propagation), with a dual-branch decoder (OFA branch aligning queries to CLIP image embeddings, CTI branch doing text cross-attention) and KL-divergence-based attention isolation masks to protect the decoder.
Key results
- On the TAO validation set, OVTR reaches 31.4 novel TETA vs. 27.8 for OVTrack (+3.6) without using novel-category proposals, with 1,732 embeddings instead of 99.4M.
- On KITTI zero-shot transfer (Car), OVTR improves MOTA from 69.8 to 71.8 over OVTrack and reduces IDs from 594 to 378.
- OVTR runs at 3.4 FPS vs. 3.1 FPS for OVTrack on TAO validation; the OVTR-Lite variant reaches 12.4 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace OVTrack with OVTR | ResNet50 tracker · TAO validation, novel categories | TETA: 27.8 -> 31.4 | Helped |
| replace OVTrack with OVTR | ResNet50 tracker · TAO test, novel categories | TETA: 24.1 -> 27.1 | Helped |
| replace OVTrack with OVTR | zero-shot tracker · KITTI validation, Car | MOTA: 69.8 -> 71.8 | Helped |
| replace OVTrack+ with OVTR | ResNet50 tracker · OVT-B, novel categories | TETA: 46.4 -> 45.5 | Hurt |
| add the OFA branch to CTI (ablation) | dual-branch decoder · TAO validation ablation | TETA: 31.2 -> 31.9 | Helped |
| add image alignment to the OFA branch (ablation) | dual-branch decoder · TAO validation ablation | TETA: 31.9 -> 33.2 | Helped |
| add both category and content isolation (ablation) | decoder protection strategies · TAO validation ablation | TETA: 32.1 -> 33.2 | Helped |
| add both category and content isolation (ablation) | decoder protection strategies · TAO validation ablation | ClsA_b: 14.6 -> 15.2; 15.6 -> 15.2 | Mixed |
| use image embeddings instead of text embeddings for alignment (ablation) | alignment head · TAO validation ablation | TETA: 31.6 -> 33.2 | Helped |
| use average text and image embeddings for alignment (ablation) | alignment head · TAO validation ablation | ClsA_n: 2.7 -> 3.2 | Helped |
| use average text and image embeddings instead of image embeddings for alignment (ablation) | alignment head · TAO validation ablation | AssocA: 34.5 -> 33.2 | Hurt |
| use OFA output O_img instead of CTI output O_txt as CIP input (ablation) | category information propagation · TAO validation ablation | TETA: 32.5 -> 33.2 | Helped |
Surprising findings
- Aligning queries to CLIP text embeddings instead of image embeddings reduces TETA by 1.6 points, and averaging both lowers base association performance.
- Novel-category classification accuracy (ClsA_n) of OVTR on TAO test is only 2.1, and base ClsA on test (14.9) is below OVTrack (16.9).
Limitations
- Ablations are run on a 40,000-image LVIS subset with reduced training, so absolute numbers differ from main results.
- Novel-category classification remains weak (ClsA_n around 2-3), and base classification on the test set trails OVTrack.
- KITTI pedestrian evaluation is inconsistent with the open-vocabulary category definition, so MOTA is less reliable for that class.
Experimental setup
Datasets: LVIS TAO KITTI OVT-B
Benchmarks: TAO open-vocabulary MOT (validation and test) KITTI zero-shot transfer OVT-B
Metrics: TETA LocA AssocA ClsA MOTA IDF1 MT ML IDs Frag
Baselines: OVTrack DeepSORT (ViLD) Tracktor++ (ViLD) MASA CenterTrack OVTrack+
Builds on: MOTR DINO-DETR CLIP OVTrack GLIP
Closest papers by meaning
- OVT-B: A New Large-Scale Benchmark for Open-Vocabulary Multi-Object Tracking (NeurIPS 2024 · similarity 0.93)
- OpenVIS: Open-vocabulary Video Instance Segmentation (AAAI 2025 · similarity 0.87)
- Foundation Model Driven Appearance Extraction for Robust Multiple Object Tracking (AAAI 2025 · similarity 0.87)
- Multiple Object Tracking as ID Prediction (CVPR 2025 · similarity 0.87)
- VastTrack: Vast Category Visual Object Tracking (NeurIPS 2024 · similarity 0.86)
- Towards Generalizable Multi-Object Tracking (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.