Research map / Detection and segmentation / Tracking
Revisiting motion information for RGB-Event tracking with MOT philosophy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CSAM, a cascade RGB-Event single object tracker that uses CEUTrack as an appearance model to propose candidates, then tracks targets and distractors via MOT-style candidate matching with motion embeddings from event data and spatial-temporal transformers.
Problem
RGB-Event single object tracking struggles to associate the true target with distractors over time, since existing RGB-E trackers rely mainly on appearance and do not exploit event-based motion cues.
Approach
Brings the MOT tracking-by-association philosophy into RGB-E SOT: an appearance model proposes candidates, then a candidate encoding module builds appearance and motion embeddings, a Spatial-Temporal Transformer Encoder models relations across frames, and a Dual-branch Transformer Decoder matches candidates to tracklets.
Key results
- CSAM-B reaches 68.1% SR and 76.7% PR on COESOT, surpassing SeqTrackv2-L256's 65.2% SR.
- On FE108, CSAM-B achieves 70.5% RSR and 97.1% RPR, improving over HRMonTrack-B by 2.0% RSR and 0.9% RPR.
- CSAM-B runs at about 53 FPS on a single RTX 3090 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AFE and MFE to the candidate encoding module (ablation) | CSAM · COESOT test | SR: 67.1 -> 68.1 | Helped |
| Add AFE and MFE to the candidate encoding module (ablation) | CSAM · COESOT test | PR: 75.9 -> 76.7 | Helped |
| Add the spatial encoder and temporal encoder (ablation) | CSAM STTE · COESOT test | SR: 66.2 -> 68.1 | Helped |
| Add the spatial encoder and temporal encoder (ablation) | CSAM STTE · COESOT test | PR: 75.3 -> 76.7 | Helped |
| Use the proposed spatial encoder instead of the original Transformer Encoder (ablation) | CSAM STTE · COESOT test | SR: 66.8 -> 68.1 | Helped |
| Use the proposed spatial encoder instead of SuperGlue (ablation) | CSAM STTE · COESOT test | SR: 67.1 -> 68.1 | Helped |
| Add the temporal encoder (ablation) | CSAM STTE · COESOT test | SR: 67.4 -> 68.1 | Helped |
| Add the spatial encoder (ablation) | CSAM STTE · COESOT test | SR: 66.6 -> 68.1 | Helped |
| Use the spatial-temporal decoder instead of the original Transformer Decoder (ablation) | CSAM DBTD · COESOT test | SR: 66.7 -> 68.1 | Helped |
| Add the motion decoder (ablation) | CSAM DBTD · COESOT test | SR: 67.7 -> 68.1 | Helped |
| Add the spatial-temporal decoder (ablation) | CSAM DBTD · COESOT test | SR: 66.4 -> 68.1 | Helped |
| Use a 2-layer STTE and DBTD rather than 1 layer (ablation) | CSAM · COESOT test | PR: 76.7 -> 76.9 | Helped |
Limitations
- Less effort on improving the efficiency of supervision signals, as acknowledged by the authors
- Supervision is partial because RGB-E datasets only provide target locations, requiring self-supervised and partial supervision losses
- Added computation and parameters relative to appearance-only tracker (106.9M vs 92.5M params)
Experimental setup
Datasets: COESOT VisEvent FE108
Benchmarks: COESOT VisEvent FE108
Metrics: success rate (SR) precision rate (PR) normalized precision rate (NPR) representative success rate (RSR) representative precision rate (RPR) FPS FLOPS
Baselines: CEUTrack ViPT SDSTrack OneTrack SeqTrackv2 HRMonTrack KeepTrack TransTAM
Builds on: CEUTrack OSTrack KeepTrack SuperGlue MAE
Closest papers by meaning
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.87)
- Temporal Adaptive RGBT Tracking with Modality Prompt (AAAI 2024 · similarity 0.87)
- SUTrack: Towards Simple and Unified Single Object Tracking (AAAI 2025 · similarity 0.86)
- Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline (CVPR 2024 · similarity 0.86)
- Cross-modulated Attention Transformer for RGBT Tracking (AAAI 2025 · similarity 0.86)
- CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.