Research map / Vision-language and multimodal / Human and embodied modeling
HuPerFlow: A Comprehensive Benchmark for Human vs. Machine Motion Estimation Comparison
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds HuPerFlow, a benchmark of human-perceived optical flow collected via an online psychophysics paradigm across ten optical flow datasets, and compares ten flow algorithms against human responses and ground truth.
Problem
Human-perceived motion in natural scenes is underexplored, and existing optical flow benchmarks only provide physical ground truth, so alignment between machine flow estimates and human motion perception cannot be measured at scale.
Approach
Collect large-scale human motion-perception vectors through an online adjustment-method psychophysics paradigm (Brownian-noise matching with flash probes) on full-frame clips from ten standard optical flow datasets, then evaluate models with EPE, correlation, partial correlation controlling for ground truth, and a Response Consistency Index.
Key results
- Collected about 38,400 response vectors at 2,400 probed locations from 42 observers over 480 sessions.
- RAFT showed the most human-like responses (r with human uv = 0.64) while being second-best to VideoFlow in ground-truth alignment (r with GT uv 0.94 for VideoFlow vs 0.93 for RAFT).
- FFV1MT and V1Attention-StageI had the highest partial correlations with human responses controlling for ground truth (0.31 and 0.23 for uv), and highest RCI values.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoFlow instead of the other evaluated optical flow algorithms | Optical flow prediction · HuPerFlow (all datasets) | EPE vs. ground truth: 1.91 | Helped |
| Use RAFT instead of the other evaluated optical flow algorithms | Optical flow prediction · HuPerFlow (all datasets) | EPE vs. human response: 7.37 | Mixed |
| Use FFV1MT instead of the other evaluated optical flow algorithms | Optical flow prediction · HuPerFlow (all datasets) | partial correlation (uv) vs. human response: 0.31 | Helped |
| Use V1Attention-StageI instead of the other evaluated optical flow algorithms | Optical flow prediction · HuPerFlow (all datasets) | partial correlation (uv) vs. human response: 0.23 | Helped |
Surprising findings
- Human-perceived flow in driving datasets aligns closely with ground truth, but in MPI Sintel and SPRING observers perceive grouped movement rather than individual locations.
- Recent CV models correlate closely with human perception yet show low partial correlation after controlling for ground truth, suggesting they do not capture human-specific perceptual biases.
Limitations
- Human-perceived flow is limited to 30 Hz display and only future-directed flow was reported.
- Only 42 observers after screening, with large individual differences (average inter-observer correlation 0.407).
- Models were evaluated with default or Sintel-fine-tuned weights, and the study does not train models on the human data.
Experimental setup
Datasets: KITTI 2015 Virtual KITTI 2 Scene Flow Driving VIPER TartanAir MPI Sintel SPRING Monkaa MHOF FlyingThings3D
Benchmarks: HuPerFlow
Metrics: endpoint error Pearson correlation partial correlation Response Consistency Index
Baselines: Farnebäck FFV1MT RAFT FlowNet2 FlowFormer VideoFlow Perceiver IO AGFlow V1Attention-StageI DualModel
Builds on: Yang et al. 2023 psychophysical perceived motion flow RAFT VideoFlow V1Attention
Closest papers by meaning
- Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception (NeurIPS 2024 · similarity 0.87)
- H-Flow: Self-supervised Human Scene Flow via Physics-inspired Joint Multi-modal Learning (NeurIPS 2026 · similarity 0.86)
- GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction (NeurIPS 2026 · similarity 0.86)
- MemFlow: Optical Flow Estimation and Prediction with Memory (CVPR 2024 · similarity 0.86)
- Aligning Human Motion Generation with Human Perceptions (ICLR 2025 · similarity 0.85)
- Flow-NeRF: Joint Learning of Geometry, Poses, and Dense Flow within Unified Neural Representations (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.