Research map / Video / Video segmentation and tracking
OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OCAI, an occlusion- and consistency-aware forward-warping video frame interpolation method that generates intermediate frames and flows, used in a teacher-student EMA semi-supervised scheme to train RAFT optical flow models.
Problem
Optical flow models are limited by scarce ground-truth labels; existing VFI models cannot reliably produce intermediate flows and forward-warping methods suffer from pixel ambiguity and holes, and prior semi-supervised approaches (RealFlow) are computationally expensive.
Approach
Forward warping with an occlusion-aware weighting mask (replacing depth weighting) resolves pixel ambiguities, forward-backward consistency fills holes in intermediate flows and fuses backward-warped frames via confidence maps, and the resulting interpolated frame/flow pairs supervise a student RAFT with an EMA teacher.
Key results
- On KITTI VFI (5->10 FPS), OCAI reaches 22.08 PSNR and 0.112 LPIPS (AlexNet), the best LPIPS with 5.3M params versus 349.5M for RIPR.
- On KITTI test, OCAI achieves 4.13 Fl-all versus 4.63 for RealFlow and 4.71 for DistractFlow, a new semi-supervised SOTA.
- On KITTI train, OCAI reaches 7.6 Fl-all vs 8.5 for RealFlow (4 EM iterations) and 11.7 for DistractFlow.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with OCAI-interpolated frames and flows | RAFT semi-supervised training · KITTI train | Fl-all: 8.5 -> 7.6 | Helped |
| train with OCAI-interpolated frames and flows | RAFT semi-supervised training · KITTI test | Fl-all: 4.63 -> 4.13 | Helped |
| use OCAI forward-warp interpolation | VFI · Sintel (12FPS →24 FPS) | LPIPS (A): 0.084 -> 0.078 | Helped |
| replace depth weighting with occlusion-aware weighting (ablation) | OCAI VFI · KITTI VFI | SSIM: 0.733 -> 0.734 | No clear effect |
| add flow-consistency hole filling (ablation) | OCAI VFI · KITTI VFI | SSIM: 0.721 -> 0.739 | Helped |
| use confidence-based fusion instead of BHF (ablation) | OCAI VFI · KITTI VFI | SSIM: 0.739 -> 0.758 | Helped |
| add EMA teacher updates (ablation) | RAFT semi-supervised training · KITTI 15 test | Fl-all: Diverged -> 10.1 | Helped |
| add confidence-based loss masking (ablation) | OCAI semi-supervised training · KITTI 15 test | Fl-all: 10.1 -> 9.7 | Helped |
| add image distraction (ablation) | OCAI semi-supervised training · KITTI 15 test | Fl-all: 9.7 -> 8.4 | Helped |
| train using OCAI-generated intermediate frames and flows (ablation) | OCAI semi-supervised training · KITTI 15 test | Fl-all: 8.4 -> 7.6 | Helped |
| use the semi-supervised optical-flow model (ablation) | OCAI VFI · Sintel (6 FPS →12 FPS) | PSNR: 25.88 -> 25.87 | No clear effect |
| use the semi-supervised optical-flow model (ablation) | OCAI VFI · KITTI (5 FPS →10 FPS) | PSNR: 22.08 -> 22.29 | Helped |
Surprising findings
- Training diverges when teacher and student share weights, so EMA is necessary for stability.
- PSNR and SSIM do not always reflect perceived VFI quality; VFIFormer has higher SSIM but worse visual quality.
Limitations
- Relies on a pre-trained RAFT for initial flow and linear-motion assumption for hole filling.
- Depth-free occlusion weighting performs comparably to depth weighting in VFI quality rather than clearly better.
- Semi-supervised gains evaluated mainly on Sintel and KITTI; VFI improvements from semi-supervised flow are small.
Experimental setup
Datasets: FlyingChairs FlyingThings3D Sintel KITTI HD1K Monkaa SlowFlow Vimeo90k
Benchmarks: Sintel KITTI SlowFlow
Metrics: PSNR SSIM LPIPS EPE Fl-all
Baselines: IFRNet-B VFIFormer RIFE EMA-VFI AMT-L SoftSplat RIPR (RealFlow) RAFT-OCTC FlowSupervisor DistractFlow RealFlow RAFT-A
Builds on: RealFlow Softmax Splatting DistractFlow Mean Teacher RAFT
Closest papers by meaning
- Imposing Consistency for Optical Flow Estimation (CVPR 2022 · similarity 0.91)
- Occlusion Aware Unsupervised Learning of Optical Flow (CVPR 2018 · similarity 0.90)
- DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework (CVPR 2025 · similarity 0.88)
- StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.86)
- Taming generative video models for zero-shot optical flow extraction (NeurIPS 2025 · similarity 0.86)
- Recurrent Partial Kernel Network for Efficient Optical Flow Estimation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.