Research map / Detection and segmentation / Tracking
Tracking objects that change in appearance with phase synchrony
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the complex-valued RNN (CV-RNN), which implements attention via neural phase synchrony, and tests it against humans and video DNNs on the FeatureTracker challenge for tracking objects that change appearance.
Problem
How biological visual systems track objects whose color and shape change over time, independently of their appearance, and whether neural synchrony can support this in artificial networks.
Approach
A complex-valued attention module in which feature magnitude encodes object appearance and phase encodes object location, trained with an added phase-synchrony loss; it is embedded in an InT RNN and compared to humans and video DNNs on a controlled tracking challenge.
Key results
- Humans were 92% accurate when color and shape changed, and 79% when both were fixed, on FeatureTracker.
- CV-RNN's error consistency with humans fell within the human inter-subject interval on 6 of 8 FeatureTracker versions, while other DNNs made errors on different videos than humans.
- On out-of-distribution color conditions, CV-RNN performed significantly better than the other models, which were close to chance accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the CV-RNN instead of the other tested DNNs | FeatureTracker · OOD colors and OOD colors plus shapes | accuracy | Helped |
| Use the CV-RNN instead of other tested models | FeatureTracker · 6/8 versions of FeatureTracker | human error consistency | Helped |
| Pretrain on the full colorspace (ablation) | CV-RNN and comparable RNN architecture · FeatureTracker | performance | Helped |
| Remove the phase synchrony loss (ablation) | CV-RNN · FeatureTracker | phase-based tracking | Hurt |
Surprising findings
- Transformer video models showed notably low accuracy even on in-distribution videos.
- The CV-RNN could not learn to use phase for tracking without an additional synchrony loss.
Limitations
- CV-RNN performs below humans on conditions with colors unseen during training.
- CV-RNN requires an additional synchrony loss to learn phase-based tracking.
- CV-RNN is more computationally expensive than a real-valued RNN with the same parameter count.
Experimental setup
Datasets: FeatureTracker (synthetic, 100,000 training videos) Kinetics400 (pre-training for some baselines) shell game (10,000 synthetic samples)
Benchmarks: FeatureTracker
Metrics: accuracy error consistency (Cohen's kappa)
Baselines: TimeSformer MViT ResNet3D MC3 InT RNN
Builds on: InT RNN PathTracker communication-through-coherence binding-by-synchrony
Closest papers by meaning
- Tracking Without Re-recognition in Humans and Machines (NeurIPS 2021 · similarity 0.89)
- Hierarchical Attentive Recurrent Tracking (NeurIPS 2017 · similarity 0.85)
- Tracking and Understanding Object Transformations (NeurIPS 2025 · similarity 0.84)
- Vision Transformers Learn Gestalt-Like Figure-Ground Cues from Natural Images (NeurIPS 2026 · similarity 0.84)
- Learning Object State Changes in Videos: An Open-World Perspective (CVPR 2024 · similarity 0.83)
- Beyond Accuracy: Tracking more like Human via Visual Search (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.