Research map / Detection and segmentation / Tracking
SynCL: A Synergistic Training Strategy with Instance-Aware Contrastive Learning for End-to-End Multi-Camera 3D Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SynCL, a training-only strategy for end-to-end multi-camera 3D tracking that adds a weight-shared self-attention-free parallel decoder with hybrid one-to-many matching, GMM-based dynamic query filtering, and instance-aware contrastive learning, reaching 58.9% AMOTA on nuScenes val.
Problem
Joint detection and tracking in query-based end-to-end 3D multi-camera trackers suffers optimization difficulties when sharing parameters, attributed to self-attention over-deduplicating object queries and self-centric attention in track queries.
Approach
During training only, remove self-attention in a weight-shared parallel cross-attention decoder, apply one-to-many matching for object queries (filtered by a GMM-based dynamic cost threshold) and one-to-one for track queries, and align object and track query embeddings of the same instance via kernel-based contrastive learning; inference uses the standard decoder unchanged.
Key results
- On nuScenes val with StreamPETR detector at 640x1600, SynCL reaches 58.9% AMOTA, outperforming OneTrack (54.8%).
- Plugged into PF-Track and MUTR3D, SynCL improves AMOTA by +3.9% and +3.7% respectively on nuScenes val at 320x800/900x1600.
- On nuScenes test, SynCL achieves 58.8% AMOTA, surpassing DORT (57.6%) and the E2E tracker DQTrack (52.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SynCL to MUTR3D | MUTR3D with DETR3D · nuScenes val | AMOTA: 32.1% -> 35.8%; +3.7% | Helped |
| Add SynCL to PF-Track | PF-Track with PETR · nuScenes val | AMOTA: 40.8% -> 44.7%; +3.9% | Helped |
| Add SynCL | Baseline#1 with PETRv2 · nuScenes val | AMOTA: 43.2% -> 45.7%; +2.5% | Helped |
| Add SynCL | Baseline#2 with StreamPETR · nuScenes val | AMOTA: 49.6% -> 51.8%; 2.2% | Helped |
| Add SynCL | StreamPETR-based tracker · nuScenes val | AMOTA: 54.8% -> 58.9%; 4.1% | Helped |
| Add instance-aware contrastive learning to hybrid matching (ablation) | PF-Track with PETR · nuScenes val | AMOTA: 42.6% -> 44.7%; 2.1% | Helped |
| Add hybrid matching (ablation) | PF-Track with PETR · nuScenes val | AMOTA: 40.8% -> 42.6%; 1.8% | Helped |
| Use dynamic GMM cost-based filtering instead of DETA (ablation) | SynCL · nuScenes val | AMOTA: 43.1% -> 44.7% | Helped |
| Use one-to-many matching in the S-decoder (ablation) | PF-Track · nuScenes val | AMOTA: 40.8% -> 38.8% | Hurt |
| Match C-decoder one-to-many queries with all GTs instead of persistent GTs (ablation) | PF-Track · nuScenes val | AMOTA: 44.7% -> 42.6% | Hurt |
| Match C-decoder one-to-many queries with new-born GTs instead of persistent GTs (ablation) | PF-Track · nuScenes val | AMOTA: 44.7% -> 41.1% | Hurt |
| Set maximum assignment constraint K to 4 instead of 5 (ablation) | SynCL · nuScenes val | AMOTA: 44.7% -> 44.3% | Hurt |
Surprising findings
- Removing self-attention from the standard decoder barely changes regression predictions and tends to generate more latent candidate boxes.
- Applying one-to-many matching in the standard (self-attention) decoder hurts performance (AMOTA 40.8% to 38.8%), attributed to self-attention's NMS-like effect.
Limitations
- Training cost increases modestly due to the parallel decoder (extra training time and GPU memory).
- Matching range and K hyperparameters (K=5, T=3) are tuned and sensitive; more frames (T>3) do not help.
- Evaluated only on nuScenes camera-based tracking.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes val nuScenes test
Metrics: AMOTA AMOTP Recall MOTA IDS FP FN NDS mAP FPS
Baselines: PF-Track MUTR3D ADATrack++ OneTrack DQTrack DORT STAR-Track DETA
Builds on: MOTR PF-Track MUTR3D DETR3D StreamPETR
Closest papers by meaning
- ADA-Track: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and Association (CVPR 2024 · similarity 0.88)
- S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking (ICML 2025 · similarity 0.86)
- Correlation-Aware Deep Tracking (CVPR 2022 · similarity 0.85)
- Time3D: End-to-End Joint Monocular 3D Object Detection and Tracking for Autonomous Driving (CVPR 2022 · similarity 0.85)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.84)
- MIMTrack: In-Context Tracking via Masked Image Modeling (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.