Research map / Detection and segmentation / Open-set and few-shot detection

Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli

NeurIPS 2024 · Detection and segmentation › Open-set and few-shot detection · approach: Unsupervised object and part discovery · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a classical motion energy model (Simoncelli & Heeger 1998), integrated with a trained segmentation head, generalizes zero-shot to random dot videos where 40 deep optical flow models fail, and matches human performance on a random dot shape identification task.

Problem

Deep optical flow models used for motion-based figure-ground segmentation do not generalize zero-shot to random dot stimuli that humans segment easily via common fate, so the computational basis of human appearance-agnostic motion segmentation is unclear.

Approach

Replace deep optical flow estimators with a fixed, neuroscience-derived motion energy model (spatio-temporal filtering in pixel space, implemented as a CNN) as the motion front end for a trained segmentation network, and compare zero-shot generalization to random dot videos against 40 optical flow models and humans.

Key results

• Motion energy segmentation reaches 60.0% IoU on random dots, more than 20 points above the second-best optical flow model (GMFlow, 38.1%), while FlowFormer++ and GMA variants fall below 10% IoU.
• On original videos FlowFormer++ achieves the best IoU (90.8%) while motion energy reaches 75.9%.
• In the human study (N=12), only the motion energy model matches or exceeds human performance on random dot shape identification; all optical flow models are below humans.

Measured effects reported in the paper

Surprising findings

• OCLR, a state-of-the-art motion segmentation model, reaches 83.8% IoU on original videos but only 2.6% on random dots.
• Training the motion energy CNN weights end-to-end with the segmentation model reduced random-dot generalization, with the original fixed weights performing best.

Limitations

• Segmentation network is intentionally small and trained only on synthetic Kubric data, so results on real-world motion segmentation are not established.
• Motion energy model is not competitive with state-of-the-art optical flow on natural scenes (75.9% vs 90.8% IoU on original videos).
• Human study uses a shape identification proxy rather than direct segmentation, and does not model internal noise or attentional lapses.

Experimental setup

Datasets: Kubric synthetic videos FlyingChairs FlyingThings3D Sintel KITTI Infinite dSprites

Benchmarks: Kubric-based i.i.d. test set random dot stimuli derived from test set human random dot shape identification study

Metrics: IoU F-Score accuracy psychometric curves

Baselines: GMFlow FlowNet2 PWC-Net GMA FlowFormer++ RAFT OCLR

Builds on: Simoncelli and Heeger motion energy model RAFT GMFlow OCLR

Closest papers by meaning

• Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception (NeurIPS 2024 · similarity 0.87)
• Vision Transformers Learn Gestalt-Like Figure-Ground Cues from Natural Images (NeurIPS 2026 · similarity 0.86)
• GEOPHYS: The Geometry of Physical Plausibility (NeurIPS 2026 · similarity 0.85)
• Contour Integration Underlies Human-Like Vision (ICML 2025 · similarity 0.85)
• HuPerFlow: A Comprehensive Benchmark for Human vs. Machine Motion Estimation Comparison (CVPR 2025 · similarity 0.85)
• Using Diffusion Priors for Video Amodal Segmentation (CVPR 2025 · similarity 0.85)