Research map / Detection and segmentation / Open-set and few-shot detection
UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UnSAMFlow, an unsupervised optical flow network that uses SAM masks for semantic augmentation, a homography-based regional smoothness loss, and a mask feature module, reaching state-of-the-art KITTI and Sintel results.
Problem
Unsupervised optical flow suffers at occlusions and motion boundaries because it lacks object-level information; prior semantic-guided methods rely on class-limited segmentation that cannot separate instances.
Approach
Use class-agnostic SAM masks as object-level cues for unsupervised flow: copy-paste semantic augmentation over SAM objects, a homography-based regional smoothness loss replacing boundary-aware smoothness (whose gradients are poor), and max-pooled mask features aggregated per SAM segment.
Key results
- Achieves 7.83% Fl-all on KITTI-2015 test, versus UPFlow 9.38% and SemARFlow 8.38%.
- Sintel final EPE on train split is 3.07 with all modules (+aug +hg +mf), versus ARFlow 3.73 with the baseline's counterpart at 3.63 in the paper's table.
- Generalization: training on KITTI and testing on Sintel yields clean EPE 5.75 with SAM versus 7.02 without.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic augmentation, homography smoothness, and mask features | UnSAMFlow · KITTI-2015 test | Fl-all (%): 9.38 -> 7.83 | Helped |
| Add semantic augmentation (ablation) | UnSAMFlow baseline · KITTI-2015 test | Fl-all (%): 9.60 -> 9.05 | Helped |
| Add homography smoothness to semantic augmentation (ablation) | UnSAMFlow · KITTI-2015 test | Fl-all (%): 9.05 -> 8.18 | Helped |
| Add mask feature module (ablation) | UnSAMFlow with semantic augmentation and homography smoothness · KITTI-2015 test | Fl-all (%): 8.18 -> 7.83 | Helped |
| Set whg=0, removing homography smoothness (ablation) | Ours (+aug +hg) · KITTI validation | Fl-all (%): 2.11 -> 2.26 | Hurt |
| Set waug=0, removing semantic augmentation (ablation) | Ours (+aug +hg) · KITTI validation | Fl-all (%): 2.11 -> 2.16 | Hurt |
| Replace homography smoothness with image-edge-aware smoothness (ablation) | Ours (+aug +hg) · KITTI validation | Fl-all (%): 2.11 -> 2.35 | Hurt |
| Replace homography smoothness with SAM-boundary-aware smoothness (ablation) | Ours (+aug +hg) · KITTI validation | Fl-all (%): 2.11 -> 2.33 | Hurt |
| Replace concatenation with residual mask-feature connection (ablation) | UnSAMFlow mask feature module · KITTI validation | Fl-all (%): 2.01 -> 2.05 | Hurt |
| Remove mask feature module (ablation) | Ours (+aug +hg +mf) · KITTI validation | Fl-all (%): 2.01 -> 2.11 | Hurt |
| Use SAM guidance during training | UnSAMFlow · KITTI→Sintel Final | EPE (px): 7.02 -> 5.75 | Helped |
| Use SAM guidance during training | UnSAMFlow · Sintel→KITTI-2015 | Fl-all (%): 8.59 -> 7.58 | Helped |
Surprising findings
- Boundary-aware (edge or SAM-boundary) smoothness losses perform worse than the homography regional smoothness, attributed to poor, local gradient landscapes.
Limitations
- Performance depends on SAM mask quality, which degrades under lighting problems, artifacts, or motion blur.
- SAM masks lack semantic class labels, so object information is incomplete.
- SAM mask generation at inference adds overhead for the mask-input setting.
Experimental setup
Datasets: KITTI raw sequences KITTI multi-view extension Sintel raw frames Sintel clean and final
Benchmarks: KITTI 2012 KITTI 2015 Sintel clean Sintel final
Metrics: EPE Fl-all Fl-noc Fl-bg Fl-fg
Baselines: ARFlow UPFlow SemARFlow UnFlow-CSS DDFlow SelFlow SimFlow UFlow SAMFlow SOF MRFlow JFS
Builds on: ARFlow SemARFlow UPFlow Segment Anything Model
Closest papers by meaning
- UnFlow: Unsupervised Learning of Optical Flow With a Bidirectional Census Loss (AAAI 2018 · similarity 0.91)
- SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model (AAAI 2024 · similarity 0.90)
- Learning by Analogy: Reliable Supervision From Transformations for Unsupervised Optical Flow Estimation (CVPR 2020 · similarity 0.90)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.88)
- M2Flow: A Motion Information Fusion Framework for Enhanced Unsupervised Optical Flow Estimation in Autonomous Driving (AAAI 2025 · similarity 0.87)
- Taming generative video models for zero-shot optical flow extraction (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.