Research map / Video / Video segmentation and tracking
MemFlow: Optical Flow Estimation and Prediction with Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MemFlow, a real-time online optical flow estimator that reads from and updates a memory buffer of past motion and context features, plus resolution-adaptive attention re-scaling, and extends it to future flow prediction.
Problem
Multi-frame optical flow methods either model long-range motion at heavy computational cost or require future frames, so they are unsuitable for real-time online estimation; flow estimators also generalize poorly across resolutions.
Approach
A memory buffer of historical context and motion keys/values is queried by attention from the current context feature to aggregate temporal motion cues into a GRU-based recurrent flow refiner, with a log-length resolution-adaptive scaling in the attention softmax; the same design is repurposed for future flow prediction.
Key results
- MemFlow reaches 0.93 EPE on Sintel clean and 2.08 on final (zero-shot), versus SKFlow's 1.22 and 2.46, and 13.7% Fl-all on KITTI-15 versus SKFlow's 15.5%.
- MemFlow-T achieves 0.85 Sintel clean, 2.06 Sintel final and 12.8% KITTI-15 Fl-all in generalization, state of the art among the compared methods.
- MemFlow runs at 5.6 fps on an A100 for 1024x436 video, and with 5 GRU iterations at 14.5 fps, while VideoFlow runs out of memory on Spring.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the MemFlow memory module to SKFlow | SKFlow · Sintel final (zero-shot) | EPE: 2.46 -> 2.08 | Helped |
| add the MemFlow memory module to SKFlow | SKFlow · KITTI-15 (zero-shot) | Fl-all: 15.5 -> 13.7 | Helped |
| replace the SKFlow feature encoder with Twins-SVT | MemFlow · Sintel clean (zero-shot) | EPE: 0.93 -> 0.85 | Helped |
| remove 2-frame pretraining (ablation) | MemFlow · Sintel clean | EPE: 1.00 -> 1.16 | Hurt |
| remove 2-frame pretraining (ablation) | MemFlow · KITTI-15 | Fl-all: 14.76 -> 16.55 | Hurt |
| disable the memory module at inference (ablation) | MemFlow · Sintel clean | EPE: 1.00 -> 1.16 | Hurt |
| increase inference memory length from 1 to 2 (ablation) | MemFlow · Sintel clean | EPE: 1.00 -> 1.02 | Hurt |
| increase inference memory length from 1 to 3 (ablation) | MemFlow · Sintel clean | EPE: 1.00 -> 1.07 | Hurt |
| train with 5-frame videos instead of 3-frame videos (ablation) | MemFlow · Sintel clean | EPE: 0.93 -> 0.97 | Hurt |
| train with 8-frame videos instead of 3-frame videos (ablation) | MemFlow · Sintel clean | EPE: 0.93 -> 0.95 | Hurt |
| add warm-start initialization (ablation) | MemFlow · Sintel clean | EPE: 0.93 -> 1.02 | No clear effect |
| add resolution-adaptive re-scaling (ablation) | MemFlow · Spring Full-HD | EPE: 0.448 -> 0.433 | Helped |
Surprising findings
- Increasing the maximum memory buffer length beyond 1 degrades performance, so a 3-frame-like setting works best.
- Warm-start has little effect once the memory module is used.
Limitations
- Long-range memory (distilled long-term memory, longer video training) did not improve results, so the benefit is mainly from short-term temporal context.
- Finetuned results trail VideoFlow-MOF on Sintel and KITTI, where multi-frame offline modeling helps dataset-specific performance.
- Video prediction results are only compared on KITTI and rely on external components (Softmax Splatting, DPT depth, inpainting).
Experimental setup
Datasets: FlyingChairs FlyingThings3D Sintel KITTI-15 HD1K Spring
Benchmarks: Sintel KITTI-15 Spring FlyingThings3D KITTI video prediction
Metrics: end-point-error Fl-all 1px outlier rate WAUC SSIM LPIPS
Baselines: RAFT GMA GMFlow GMFlowNet SKFlow MatchFlow FlowFormer++ EMD-L TransFlow VideoFlow OFNet CroCo-Flow
Builds on: RAFT SKFlow GMA Twins-SVT Softmax Splatting
Closest papers by meaning
- StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.90)
- DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework (CVPR 2025 · similarity 0.88)
- MESSENGER: Memory-Enhanced Sequential Scene Flow Estimation via Autoregressive Next-Frame Forecasting (NeurIPS 2026 · similarity 0.88)
- Recurrent Partial Kernel Network for Efficient Optical Flow Estimation (AAAI 2024 · similarity 0.88)
- EDCFlow: Exploring Temporally Dense Difference Maps for Event-based Optical Flow Estimation (CVPR 2025 · similarity 0.87)
- Efficient Meshflow and Optical Flow Estimation from Event Cameras (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.