Research map / Detection and segmentation / Open-set and few-shot detection
SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAMFlow, which embeds a frozen SAM image encoder into FlowFormer and adapts it to optical flow via a Context Fusion Module and a Context Adaption Module, reducing fragmented flow estimates.
Problem
Optical flow estimators rely on local clues and ignore object integrity, producing fragmented motion estimates for the same object under occlusion and complex lighting/textures; training data for flow is also scarce.
Approach
Use the frozen SAM image encoder's object-level perception as the context encoder inside FlowFormer, and bridge the task mismatch with a Context Fusion Module (fuses SAM and FlowFormer features via residual convolutions) and a Context Adaption Module (learned task-specific embedding tokens with two-way attention).
Key results
- On Sintel train, SAMFlow reaches 0.86 clean / 2.10 final EPE and 3.55 EPE / 12.32 F1-all on KITTI-15 train, surpassing FlowFormer by 8.5%/9.9% and 13.2%/16.3%.
- On Sintel test clean pass, SAMFlow reports 1.00 EPE, ranking #1 among two-frame methods at the time of submission.
- On KITTI-15 test, SAMFlow achieves 4.49 F1-all, ahead of FlowFormer (4.68) and FlowFormer++ (4.52).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAMFlow instead of FlowFormer++ | Optical flow estimation · Sintel(train) C+T | clean EPE: 0.90 -> 0.87 | Helped |
| Use SAMFlow instead of FlowFormer++ | Optical flow estimation · Sintel(test) C+T+S+K+H | final EPE: 1.94 -> 2.08 | Hurt |
| Use SAMFlow instead of FlowFormer++ | Optical flow estimation · KITTI-15(test) | F1-all: 4.52 -> 4.49 | Helped |
| Add SAM encoder in place of FlowFormer encoder (ablation) | SAMFlow · Sintel(train) | clean EPE: 0.94 -> 0.89 | Helped |
| Add SAM encoder in place of FlowFormer encoder (ablation) | SAMFlow · KITTI-15(train) | EPE: 4.09 -> 4.11 | Hurt |
| Add CFM to fuse FlowFormer and SAM features (ablation) | SAMFlow · KITTI-15(train) | EPE: 4.11 -> 3.83 | Helped |
| Add CAM without CFM (ablation) | SAMFlow · KITTI-15(train) | EPE: 4.09 -> 3.81 | Helped |
| Add both CFM and CAM (ablation) | SAMFlow · KITTI-15(train) | EPE: 4.09 -> 3.44 | Helped |
| Add both CFM and CAM (ablation) | SAMFlow · Sintel(train) | clean EPE: 0.94 -> 0.87 | Helped |
| Use MobileSAM encoder instead of SAM-H (ablation) | SAMFlow · Sintel(train) | final EPE: 2.11 -> 2.19 | Hurt |
| Use MobileSAM encoder instead of SAM-H (ablation) | SAMFlow · KITTI-15(train) | EPE: 3.44 -> 3.78 | Hurt |
| Replace SAM encoder with ViT (ablation) | SAMFlow · Sintel(train) | clean EPE: 0.87 -> 1.08 | Hurt |
Surprising findings
- Plain ViT, MAE and DINO encoders do not help optical flow, and SAM's feature alone gives limited gains on real-world KITTI data without task adaptation.
- MobileSAM outperformed SAM-B on the Sintel final pass, which the authors attribute to architectural differences.
Limitations
- Adding the SAM encoder increases computation, with runtime depending on SAM encoder scale.
- Plain SAM features lose fine spatial detail and need task-specific adaptation to be useful for flow.
- Requires modifying SAM's positional and relative positional encodings to support smaller input resolutions.
Experimental setup
Datasets: FlyingThings Sintel KITTI-15 HD1K
Benchmarks: Sintel KITTI-15
Metrics: EPE F1-all occluded-area EPE runtime
Baselines: FlowFormer FlowFormer++ GMFlow SKFlow GMFlowNet RAFT GMA CRAFT FM-RAFT S-Flow
Builds on: FlowFormer Segment Anything Model Perceiver IO MobileSAM
Closest papers by meaning
- UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model (CVPR 2024 · similarity 0.90)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.88)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.87)
- DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework (CVPR 2025 · similarity 0.87)
- MemFlow: Optical Flow Estimation and Prediction with Memory (CVPR 2024 · similarity 0.86)
- Flow-NeRF: Joint Learning of Geometry, Poses, and Dense Flow within Unified Neural Representations (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.