Research map / Video / Video segmentation and tracking
TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TimeTracker, an event-based video frame interpolation framework that segments the scene into appearance-similar regions and tracks them continuously with events to get dense any-time optical flow, plus a new real-world high-speed dataset (CHMD).
Problem
Event-guided video frame interpolation under non-linear motion, where sparse event-based flow and dense image-event feature fusion produce motion errors and temporal/spatial misalignment.
Approach
Transforms any-time dense optical flow estimation into local feature tracking: SLIC superpixels from images act as tracking templates, and a transformer-based point tracker driven by event voxels follows each region's continuous trajectory, followed by global flow optimization and frame refinement.
Key results
- On GoPro, TimeTracker reaches 37.13 PSNR / 0.962 SSIM, versus 37.02 / 0.959 for TimeLens-XL.
- On SNU-FILM Extreme, TimeTracker reports 29.27 PSNR vs 27.93 for TimeLens-XL.
- On BS-ERGB at 15 skips, TimeTracker reports 27.69 PSNR vs 24.77 for TimeLens-XL; ablation shows removing superpixel or point tracking drops GoPro PSNR to 30.62 and 27.15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TimeTracker instead of TimeLens-XL | Video frame interpolation · GoPro | PSNR: 37.02 -> 37.13 | Helped |
| Use TimeTracker instead of TimeLens-XL | Video frame interpolation · SNU-FILM 15skips | PSNR: 30.95 -> 32.86 | Helped |
| Use TimeTracker instead of TimeLens-XL | Video frame interpolation · BS-ERGB 15skips | PSNR: 24.77 -> 27.69 | Helped |
| Add superpixel segmentation (ablation) | TimeTracker · GoPro | PSNR: 30.62 -> 37.47 | Helped |
| Add point tracking (ablation) | TimeTracker · GoPro | PSNR: 27.15 -> 37.47 | Helped |
| Add global optical flow optimization (ablation) | TimeTracker · GoPro | PSNR: 32.43 -> 37.47 | Helped |
| Add frame refinement (ablation) | TimeTracker · GoPro | PSNR: 35.29 -> 37.47 | Helped |
Surprising findings
- Frame-based SuperSlomo and PerVFI, and even the event-based TimeLens, degrade sharply at higher skip counts while TimeTracker stays more stable.
- Voxel bin size optimum differs by dataset: 0.005s best on BS-ERGB, but on CHMD finer bins help because fast motion produces higher event density.
Limitations
- Tracking-based motion estimation struggles with dynamic textures such as fluids, where features are temporally discontinuous.
- Relies on synthesized occlusion regions handled by a refinement network, with a fallback to synthesis in those areas.
Experimental setup
Datasets: TAP-Vid MultiFlow GoPro SNU-FILM BS-ERGB
Benchmarks: GoPro SNU-FILM BS-ERGB CHMD
Metrics: PSNR SSIM
Baselines: SuperSlomo PerVFI VFIT-B TimeLens CBMNet SuperFast TimeLens-XL
Builds on: RAFT TAP-Vid point tracking TimeLens SLIC E-RAFT
Closest papers by meaning
- Video Frame Interpolation via Direct Synthesis with the Event-based Reference (CVPR 2024 · similarity 0.89)
- Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation (CVPR 2025 · similarity 0.88)
- TTA-EVF: Test-Time Adaptation for Event-based Video Frame Interpolation via Reliable Pixel and Sample Estimation (CVPR 2024 · similarity 0.87)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.87)
- Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better (CVPR 2025 · similarity 0.87)
- E-Motion: Future Motion Simulation via Event Sequence Diffusion (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.