Research map / Video / Action recognition
Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Motion Perceiver, a model that learns from patch-level optical flow of natural RGB videos via flow snapshot neurons (slot attention) and motion-invariant neurons, and generalizes zero-shot to point-light biological motion stimuli, outperforming video models on a new 62,656-stimulus BMP benchmark.
Problem
AI action-recognition models generalize poorly to biological motion perception (point-light displays) without task-specific training, whereas humans do so readily; no systematic benchmark existed for evaluating this generalization.
Approach
Compute dense patch-level optical flow over all frames using frozen DINO feature similarities, learn prototypical flow snapshot neurons via slot attention with a contrastive walk loss, and add motion-invariant neurons that aggregate directional flow magnitudes independent of temporal order.
Key results
- MP reaches 69.00% top-1 on J-6P versus at most ~15.7% for the best external baseline (TwoStream-CNN) in the supplementary comparison, and outperforms all baselines on BMP conditions (up to 29% improvement claimed).
- MP achieves 96.45% on RGB, 69.00% on J-6P and 49.68% on SP-8P-1LT, while humans reach about 88% overall across the three modalities.
- MP correlates more strongly with human performance across BMP conditions than all competing AI baselines, and shows higher error-pattern consistency with humans at the trial level.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the motion-invariance pathway (ablation) | MP · RGB-R | top-1 accuracy (%): 49.58% -> 62.32% | Helped |
| add the motion-invariance pathway (ablation) | MP · RGB-S | top-1 accuracy (%): 38.03% -> 61.34% | Helped |
| add the flow-snapshot pathway (ablation) | MP · J-6P | top-1 accuracy (%): 42.52 -> 69.00 | Helped |
| use multi-scale rather than single-scale flow-snapshot neurons (ablation) | MP · J-6P | top-1 accuracy (%): 58.43 -> 69.00 | Helped |
| use flow-snapshot activations rather than raw optical flows for fusion and classification (ablation) | MP · J-6P | top-1 accuracy (%): 54.67 -> 69.00 | Helped |
| apply the small-flow threshold γ (ablation) | MP · J-6P | top-1 accuracy (%): 42.38 -> 69.00 | Helped |
| avoid random frame shuffling and reversal during training (ablation) | MP · J-6P | top-1 accuracy (%): 58.32 -> 69.00 | Helped |
| use time embeddings (ablation) | MP · SP-8P-1LT | top-1 accuracy (%): 47.12 -> 49.68 | Helped |
| include the contrastive-walk loss L_slot (ablation) | MP · J-6P | top-1 accuracy (%): 67.84 -> 69.00 | Helped |
| use multiple reference frames for optical-flow calculation (ablation) | MP · J-6P | top-1 accuracy (%): 58.60 -> 69.00 | Helped |
| use patch-level DINO optical flow instead of downscaled pixel-level flow (ablation) | MP · J-6P | top-1 accuracy (%): 12.6% -> 69.0% | Helped |
| use DINO-ViT rather than ResNet50 as the feature extractor (ablation) | MP · J-6P | top-1 accuracy (%): 40.34% -> 69.00% | Helped |
Surprising findings
- Training with random shuffling and reversal of frames as augmentation impaired generalization on Joint and SP videos.
- Feeding pixel-level optical flow (GMFlow) to the baselines yields near-chance accuracy (about 10%) on Joint videos, while MP's patch-level flow does far better.
Limitations
- Limited to 10 action classes in the BMP dataset and to point-light displays; does not model attention or top-down influences.
- Absolute accuracy remains below human performance in many BMP conditions; En-MP variant needed to approach human level.
- MP is larger (57.5M trainable parameters) than several baselines, though the authors argue performance is not due to size.
Experimental setup
Datasets: NTU RGB+D 120 NTU RGB+D 60 NW-UCLA Kinetics 400 ImageNet
Benchmarks: BMP dataset (24 conditions, 62,656 stimuli) NTU RGB+D 60 Joint videos NW-UCLA Joint videos
Metrics: top-1 accuracy correlation with human performance error pattern consistency
Baselines: ResNet3D I3D X3D R(2+1)D SlowFast MViT E2-S-X3D VideoMAE TwoStream-CNN
Builds on: slot attention DINO contrastive walk loss FFCV
Closest papers by meaning
- Modeling Human Visual Motion Processing with Trainable Motion Energy Sensing and a Self-attention Network (NeurIPS 2023 · similarity 0.91)
- Leveraging Saccades to Learn Smooth Pursuit: A Self-Organizing Motion Tracking Model Using Restricted Boltzmann Machines (AAAI 2017 · similarity 0.88)
- Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli (NeurIPS 2024 · similarity 0.87)
- HuPerFlow: A Comprehensive Benchmark for Human vs. Machine Motion Estimation Comparison (CVPR 2025 · similarity 0.87)
- Reanimating Images using Neural Representations of Dynamic Stimuli (CVPR 2025 · similarity 0.86)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.