Research map / Video / Video segmentation and tracking
StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StreamFlow, a multi-frame optical flow method that predicts successive unidirectional flows in one non-overlapping in-batch pass (SIM pipeline), with an integrative spatio-temporal encoder module (ISC) and a global temporal regressor decoder (GTR).
Problem
Recursive multi-frame optical flow methods repeatedly compute overlapping pairwise flows on video, causing large computational redundancy, while occlusions remain hard for two-frame methods.
Approach
A Streamlined In-batch Multi-frame pipeline splits frames into non-overlapping groups so each frame's features and correlations are computed once and all unidirectional flows in a group are predicted in one forward pass; parameter-free spatio-temporal attention in the Twins encoder (ISC) and a temporal transformer in the decoder (GTR) model temporal cues within this constraint.
Key results
- Sintel test: clean 0.87 vs 1.00 for VideoFlow-BOF (C+T setting), final 2.11 vs 2.19 in train split; with C+T+S+K+H, Sintel final 1.04 vs 1.00 for VideoFlow-BOF is not better, while clean 0.38 and KITTI Fl-all 4.24 vs 4.44 for VideoFlow-BOF.
- KITTI-15 test Fl-all improves from 4.44 (VideoFlow-BOF) to 4.24 in the C+T+S+K+H setting.
- Claimed 63.82% speed improvement over previous multi-frame methods, with runtime comparable to two-frame methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the SIM pipeline instead of the recursive pipeline (ablation) | StreamFlow · Sintel | runtime (ms): 122.18 -> 84.59 | Helped |
| Add temporal attention (ablation) | StreamFlow encoder · Sintel clean | EPE: 1.03 -> 0.96 | Helped |
| Add temporal attention (ablation) | StreamFlow encoder · KITTI | Fl-all: 14.70 -> 14.96 | Hurt |
| Use pseudo 3D convolution (ablation) | StreamFlow encoder · Sintel clean | EPE: 1.03 -> 1.05 | Hurt |
| Use 3D convolution (ablation) | StreamFlow encoder · Sintel clean | EPE: 1.03 -> 0.98 | Helped |
| Add ISC (ablation) | StreamFlow encoder · Sintel clean | EPE: 1.01 -> 0.93 | Helped |
| Widen the baseline network with additional parameters (ablation) | StreamFlow with ISC · Sintel clean | EPE: 0.97 -> 0.98 | Hurt |
| Add GTR (ablation) | StreamFlow decoder · Sintel clean | EPE: 0.97 -> 0.93 | Helped |
| Use 4 rather than 3 input frames (ablation) | StreamFlow · Sintel clean | EPE: 0.93 -> 0.87 | Helped |
| Use StreamFlow instead of VideoFlow-BOF | optical flow estimation · Sintel test clean | EPE: 1.00 -> 1.04 | Hurt |
| Use StreamFlow instead of VideoFlow-BOF | optical flow estimation · Sintel test final | EPE: 1.71 -> 1.87 | Hurt |
| Use StreamFlow instead of TransFlow | optical flow estimation · KITTI-15 test | Fl-all: 4.32 -> 4.24 | Helped |
Surprising findings
- Simply adding parameters to the two-frame baseline gives only minor gains, suggesting the ISC/GTR improvements stem from the temporal modules rather than capacity.
- Common temporal modeling alternatives (temporal attention, pseudo-3D conv, 3D conv) in the encoder give smaller or inconsistent improvements, which the authors attribute to limited flow data for training temporal modules from scratch.
Limitations
- Number of frames limited to 3 or 4 due to GPU memory constraints
- Temporal modules in the encoder show limited gains when trained from scratch on limited flow data
Experimental setup
Datasets: FlyingThings FlyingChairs Sintel KITTI HD1K
Benchmarks: Sintel KITTI-15 DAVIS
Metrics: end-point error Fl-all runtime (ms) parameters
Baselines: RAFT GMA SKFlow FlowFormer TransFlow VideoFlow-BOF MFCFlow MFRFlow Twins-SKFlow
Builds on: RAFT VideoFlow SKFlow Twins GMA
Closest papers by meaning
- MemFlow: Optical Flow Estimation and Prediction with Memory (CVPR 2024 · similarity 0.90)
- DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework (CVPR 2025 · similarity 0.90)
- Taming generative video models for zero-shot optical flow extraction (NeurIPS 2025 · similarity 0.90)
- MESSENGER: Memory-Enhanced Sequential Scene Flow Estimation via Autoregressive Next-Frame Forecasting (NeurIPS 2026 · similarity 0.88)
- Recurrent Partial Kernel Network for Efficient Optical Flow Estimation (AAAI 2024 · similarity 0.88)
- FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.