Research map / Video / Video segmentation and tracking
ETAP: Event-based Tracking of Any Point
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ETAP, the first event-only tracking-any-point model, trained on a new synthetic event dataset (EventKubric) with a contrastive feature-alignment loss for motion-robust features.
Problem
Point tracking with event cameras is hard because synthetic event training data is immature and event features depend on motion direction, breaking the appearance correlation used by TAP trackers.
Approach
A transformer-based CoTracker-style tracker operating on event stacks, trained on a Kubric+FILM+ESIM event dataset, with a time-inversion contrastive feature-alignment loss that enforces motion-invariant descriptors.
Key results
- On EventKubric/EVIMO2 TAP, ETAP improves average Jaccard by 136% over E2VID+CoTracker on EventKubric (0.546 vs 0.236 AJ).
- On EDS feature tracking, ETAP reaches feature age 0.704, a ~20% gain over the best event-only method DDFT E2VID (0.589).
- ETAP (0.704 FA on EDS) surpasses the best events-and-frames method FE-TAP (0.676) by 4.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add feature-alignment loss (ablation) | ETAP · EVIMO2 | AJ: 0.655 -> 0.661 | Helped |
| Add feature-alignment loss (ablation) | ETAP · EventKubric | AJ: 0.556 -> 0.546 | Hurt |
| Add feature-alignment loss (ablation) | ETAP · E2D2 | AJ: 0.268 -> 0.308 | Helped |
| Add feature-alignment loss (ablation) | ETAP · EDS | Feature Age: 0.698 -> 0.704 | Helped |
| Add feature-alignment loss (ablation) | ETAP · EC | Feature Age: 0.885 -> 0.888 | Helped |
| Increase base render frame rate from 12 to 48 (ablation) | MOVI-F-trained tracker · EDS | Feature Age: 0.618 -> 0.617 | Helped |
| Use varying scene dynamics with panning motion (ablation) | MOVI-F-trained tracker · EDS | Feature Age: 0.598 -> 0.617 | Helped |
| Use random contrast thresholds U(0.16, 0.34) instead of fixed 0.2 (ablation) | MOVI-F-trained tracker · EDS | Feature Age: 0.598 -> 0.627 | Helped |
| Add Gaussian noise augmentation (ablation) | MOVI-F-trained tracker · EDS | Feature Age: 0.598 -> 0.631 | Helped |
| Use event stacks instead of voxel grids (ablation) | MOVI-F-trained tracker · EDS | Feature Age: 0.592 -> 0.598 | Helped |
| Train on EventKubric rather than MultiFlow (ablation) | ETAP · EDS | Feature Age: 0.221 -> 0.646 | Helped |
| Increase resolution from 256×256 to 512×512 (ablation) | MOVI-F-trained tracker · EDS | Feature Age: 0.598 -> 0.659 | Helped |
Surprising findings
- On the E2D2 fidget spinner sequence, frame-based CoTracker fails, while event-based tracking succeeds at higher temporal resolution.
Limitations
- Relies on query times during scene motion; track features initialized without motion do not capture appearance well.
- Monochrome event cameras cannot use color for appearance correspondence.
Experimental setup
Datasets: EventKubric MOVi-F MultiFlow EVIMO2 E2D2 EDS EC
Benchmarks: EventKubric EVIMO2 E2D2 EDS EC
Metrics: average Jaccard average pixel within threshold occlusion accuracy feature age expected feature age
Baselines: CoTracker E2VID+CoTracker ICP EKLT DDFT FE-TAP EM-ICP HASTE DDFT E2VID MultiFlow
Builds on: CoTracker TAP-Vid Vid2e Kubric FILM
Closest papers by meaning
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.87)
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D (NeurIPS 2024 · similarity 0.87)
- TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion (CVPR 2025 · similarity 0.86)
- Is This Tracker On? A Benchmark Protocol for Dynamic Tracking (NeurIPS 2025 · similarity 0.86)
- TAPTRv2: Attention-based Position Update Improves Tracking Any Point (NeurIPS 2024 · similarity 0.86)
- TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.