Research map / Architectures and efficiency / RNNs and normalization
Recurrent Partial Kernel Network for Efficient Optical Flow Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RPKNet, a small recurrent encoder-decoder optical flow model using Partial Kernel Convolution and Separable Large Kernels that generalizes best on Spring without finetuning with far fewer parameters and FLOPs.
Problem
Optical flow models have grown large and complex, which hurts efficiency and generalization when optical flow training data is scarce.
Approach
A spatial recurrent encoder that shares one block across feature scales using Partial Kernel Convolution (kernel sub-sampled per scale with variable channel counts), plus Separable Large Kernel modules built from only 1D large convolutions with structural re-parameterization skips, and SLK GRUs in a RAFT-style pyramid decoder.
Key results
- Ranks first among published methods on the Spring benchmark without finetuning, improving results by over 10% with an order of magnitude fewer FLOPs and over four times less memory than the next published method.
- Outperforms RAFT, FlowFormer-S, and FlowFormer on cross-dataset train-set generalization (Sintel clean 1.31 train, KITTI 2015 4.64 train) with the fewest parameters (2.8M) and FLOPs (1.0T).
- Ablation: PKConv encoder improves Sintel final from 2.68 (ResNet) to 2.60 and KITTI 2015 from 5.40 to 4.23; kernel size 23 performs best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the PKConv encoder with ResNet (ablation) | RPKNet · Sintel clean | EPE: 1.40 -> 1.28 | Helped |
| replace 2 SLKGRU units with 2 ConvGRU units (ablation) | RPKNet · Sintel clean | EPE: 1.39 -> 1.28 | Helped |
| replace SLKGRU units with SLK modules without GRU (ablation) | RPKNet · Sintel clean | EPE: 1.35 -> 1.28 | Helped |
| use 1 SLKGRU unit instead of 2 (ablation) | RPKNet · Sintel clean | EPE: 1.35 -> 1.28 | Helped |
| use kernel size 3 instead of 23 (ablation) | RPKNet SLK · Sintel clean | EPE: 1.39 -> 1.28 | Helped |
| use kernel size 7 instead of 23 (ablation) | RPKNet SLK · Sintel clean | EPE: 1.29 -> 1.28 | Helped |
| use kernel size 15 instead of 23 (ablation) | RPKNet SLK · Sintel clean | EPE: 1.34 -> 1.28 | Helped |
| use kernel size 31 instead of 23 (ablation) | RPKNet SLK · Sintel clean | EPE: 1.32 -> 1.28 | Helped |
| add structural re-parameterization skip connections (ablation) | RPKNet · Sintel clean | EPE: 1.32 -> 1.28 | Helped |
| use a joint encoder instead of individual feature and context encoders (ablation) | RPKNet · Sintel clean | EPE: 1.34 -> 1.28 | Helped |
| use RPKNet instead of DIP | optical flow estimation · KITTI 2015 test | Fl-All: 4.21 -> 4.64 | Hurt |
| use RPKNet instead of SKFlow | optical flow estimation · Sintel clean test | EPE: 1.22 -> 1.31 | Hurt |
Surprising findings
- Larger models do not necessarily generalize better on limited optical flow data, with the small model outperforming MatchFlow which uses extra MegaDepth pretraining.
Limitations
- Large kernel operations are not well optimized in current software and hardware.
- Larger methods such as FlowFormer still outperform it on standard benchmarks when computation is not a concern.
- Finetuned Sintel performance trails larger models that undergo extensive finetuning.
Experimental setup
Datasets: FlyingChairs FlyingThings3D Sintel KITTI 2015 HD1K Spring
Benchmarks: Spring Sintel KITTI 2015
Metrics: EPE Fl-All 1px error FLOPs memory parameters inference time
Baselines: RAFT FlowFormer FlowFormer-S MatchFlow GMA SKFlow GMFlowNet MS-RAFT+ PWCNet Flow1D DIP
Builds on: RAFT ConvGRU SKFlow Group Normalization structural re-parameterization
Closest papers by meaning
- StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.88)
- Context-Aware Iteration Policy Network for Efficient Optical Flow Estimation (AAAI 2024 · similarity 0.88)
- MemFlow: Optical Flow Estimation and Prediction with Memory (CVPR 2024 · similarity 0.88)
- FlowDiffuser: Advancing Optical Flow Estimation with Diffusion Models (CVPR 2024 · similarity 0.87)
- Taming generative video models for zero-shot optical flow extraction (NeurIPS 2025 · similarity 0.87)
- DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.