Research map / Video / Video segmentation and tracking

Taming generative video models for zero-shot optical flow extraction

NeurIPS 2025 · Video › Video segmentation and tracking · approach: Stochastic video prediction · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that frozen LRAS generative video model can be prompted zero-shot for optical flow via KL-tracing, a counterfactual perturbation method comparing logit distributions, competitive with supervised and unsupervised flow methods on TAP-Vid DAVIS and Kubric.

Problem

Zero-shot optical flow extraction from frozen self-supervised video generators without fine-tuning, avoiding synthetic-data supervision and photometric-loss heuristics.

Approach

Inject a small white Gaussian tracer into the first frame, run an LRAS autoregressive next-frame predictor with partial second-frame patches, and locate flow as the patch with maximal KL divergence between clean and perturbed predictive logit distributions, which avoids sampling noise in RGB differences.

Key results

• LRAS with KL-tracing reaches AJ 44.16 and AD 11.18 on TAP-Vid DAVIS (TAP-Vid First), outperforming RAFT, SEA-RAFT, Doduo, SMURF and CWM on that setting.
• On the DAVIS 3% subset, KL-tracing reduces endpoint error to 5.08 versus 8.48 for RGB differencing with LRAS and 74.80 for Stable Video Diffusion.
• On Kubric, KL-tracing gets AJ 65.49 and AD 5.06, the best AD among the compared methods, though SMURF and SEA-RAFT have higher AJ.

Measured effects reported in the paper

Surprising findings

• Cosmos autoregressive model fails under all partial-frame conditioning settings, with the perturbation not propagated even when the full second frame is provided.
• Deterministic CWM blurs predictions, so its flow accuracy is poor on DAVIS (AJ 15.00) despite strong occlusion accuracy.

Limitations

• Point-by-point propagation of perturbations makes inference slower than real-time dense flow baselines.
• Cosmos and SVD variants perform poorly, so the method depends on a specific architecture (LRAS) with rare properties.
• Two-frame setting is used for comparison, and multi-frame trackers like AllTracker are stronger on DAVIS.

Experimental setup

Datasets: TAP-Vid DAVIS TAP-Vid Kubric

Benchmarks: TAP-Vid DAVIS TAP-Vid Kubric

Metrics: AJ AD <delta_avg OA EPE

Baselines: RAFT SEA-RAFT Doduo SMURF DiffTrack CWM Stable Video Diffusion Cosmos CoTracker-v3 AllTracker

Builds on: Counterfactual World Model LRAS VideoShop

Closest papers by meaning

• StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.90)
• Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals (NeurIPS 2025 · similarity 0.88)
• Zero-Shot Monocular Scene Flow Estimation in the Wild (CVPR 2025 · similarity 0.88)
• ZeroFlow: Scalable Scene Flow via Distillation (ICLR 2024 · similarity 0.88)
• Optical-Flow Guided Prompt Optimization for Coherent Video Generation (CVPR 2025 · similarity 0.87)
• Recurrent Partial Kernel Network for Efficient Optical Flow Estimation (AAAI 2024 · similarity 0.87)