Research map / Video / Video segmentation and tracking
M2Flow: A Motion Information Fusion Framework for Enhanced Unsupervised Optical Flow Estimation in Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M2Flow, an unsupervised multi-frame optical flow network built on ARFlow that propagates motion information across four frames via an MIP module and adds two augmentations (CFA, CLDA) to better handle occlusions.
Problem
Unsupervised optical flow estimation fails in occluded regions, and existing multi-frame unsupervised methods rely on a constant velocity assumption and lack multi-frame occlusion augmentations, which is problematic in autonomous driving.
Approach
Replace the constant-velocity assumption with a Motion Information Propagation module that fuses learned inter-frame motion residuals from four frames, jointly estimating bidirectional flows, plus consecutive-frame (mask-based) and chain-based large-displacement self-supervision augmentations.
Key results
- On KITTI test, M2Flow reaches EPE=1.2 on KITTI-2012 and F1-all=7.37% on KITTI-2015, beating UPFlow (EPE=1.4, F1-all=9.38%) and the semantic-input UnSAMFlow (F1-all=7.83%).
- Ablation on KITTI-2015 train: adding MIP, CLDA and CFA lowers F1-all from 9.42 to 7.12 and EPE-occ from 6.71 to 4.31.
- Trained on Cityscapes and tested on KITTI-2015 train without fine-tuning, M2Flow achieves F1-all 9.33 vs 13.74 for ARFlow.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use M2Flow instead of UPFlow | unsupervised optical flow estimation · KITTI-2012 test | EPE: 1.4 -> 1.2 | Helped |
| use M2Flow instead of UPFlow | unsupervised optical flow estimation · KITTI-2015 test | Fl-all: 9.38% -> 7.37% | Helped |
| use M2Flow instead of UnSAMFlow | unsupervised optical flow estimation · Sintel Clean test all | EPE: 3.93 -> 3.38 | Helped |
| use M2Flow instead of UnSAMFlow | unsupervised optical flow estimation · Sintel Final test all | EPE: 5.20 -> 5.01 | Helped |
| add Motion Information Propagation (MIP) (ablation) | M2Flow · KITTI-2015 train | F1-all: 8.19 -> 7.12 | Helped |
| add consecutive frames augmentation (CFA) (ablation) | M2Flow with MIP and CLDA · KITTI-2015 train | F1-all: 8.01 -> 7.12 | Helped |
| add chain-based large displacement augmentation (CLDA) (ablation) | M2Flow with MIP and CFA · KITTI-2015 train | F1-all: 8.19 -> 7.12 | Helped |
| add CFA (ablation) | M2Flow with MIP and CLDA · KITTI-2015 train | EPE-noc: 1.44 -> 1.74 | Hurt |
| add CLDA (ablation) | M2Flow with MIP and CFA · KITTI-2015 train | EPE-occ: 5.16 -> 4.31 | Helped |
| replace constant-velocity motion propagation with MIP (ablation) | 4-frame M2Flow · KITTI-2015 train | F1-all: 7.77 -> 7.12 | Helped |
| increase input from 4 frames to 5 frames (ablation) | M2Flow with MIP, CFA, and CLDA · KITTI-2015 train | F1-all: 7.12 -> 7.18 | Hurt |
| increase input from 4 frames to 5 frames (ablation) | M2Flow with MIP, CFA, and CLDA · KITTI-2015 train | EPE-occ: 4.31 -> 4.02 | Helped |
Surprising findings
- A 5-frame variant of M2Flow improves occlusion error further but has worse overall EPE and F1-all than the 4-frame model, suggesting more frames need appropriate fusion.
Limitations
- Slightly higher non-occluded error (F1-noc 5.73) than semantic-input methods SemARFlow and UnSAMFlow.
- Depends on an off-the-shelf segmentation model and SAM2 for CFA masks on KITTI and Sintel, adding preprocessing dependence.
- Naive 5-frame extension does not improve overall accuracy over 4 frames.
Experimental setup
Datasets: KITTI 2012 KITTI 2015 Sintel Cityscapes
Benchmarks: KITTI 2012 KITTI 2015 Sintel
Metrics: EPE Fl-all Fl-noc EPE-occ
Baselines: ARFlow UPFlow SemARFlow UnSAMFlow SelFlow Back2Future UFlow RAFT VideoFlow-MOF MemFlow-T PWC-Fusion PWC-Net+
Builds on: ARFlow UPFlow SemARFlow UnSAMFlow SelFlow
Closest papers by meaning
- UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model (CVPR 2024 · similarity 0.87)
- OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation (CVPR 2024 · similarity 0.86)
- DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework (CVPR 2025 · similarity 0.86)
- SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model (AAAI 2024 · similarity 0.86)
- StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.85)
- MemFlow: Optical Flow Estimation and Prediction with Memory (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.