Research map / 3D vision / Depth and stereo
ADFactory: An Effective Framework for Generalizing Optical Flow with NeRF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADF, a framework that builds an optical flow training dataset from monocular photos by reconstructing scenes with Zip-NeRF, rendering flow labels between camera poses, and filtering labels with confidence, SSIM and depth-consistency masks.
Problem
Optical flow methods generalize poorly to real-world scenes because large-scale real-world flow datasets are lacking, and existing self-supervised photometric losses are indirect and suffer from occlusions, giving fuzzy results.
Approach
Use NeRF (Zip-NeRF) reconstructions of casually captured static scenes to render RGB, depth and camera-pose-based optical flow labels, then filter erroneous labels using a midpoint-depth-based radiation field confidence mask, SSIM warp error mask and multi-view depth consistency mask, and train flow networks directly on the filtered labels.
Key results
- Scale-flow trained on ADF58 reaches K15 Flepe 3.88 and K12 Flepe 1.59, outperforming MDFlow-fast zero-shot results on KITTI (e.g., MDFlow Sintel-trained K15 Flepe 10.05).
- On DAVIS zero-shot real-world evaluation, RAFT trained on ADF58 gets foreground SSIM loss 0.22 vs. 0.29 for KITTI-finetuned RAFT.
- Mconf (radiation field confidence) gives the largest single-mask gain in the RAFT ablation; combining all masks yields K15 Flepe 4.19 / Fall 14.15 and Midd-A Flepe 0.28.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the ADF58 data factory training scheme instead of KITTI fine-tuning | Scale-flow · DAVIS real-world | Sfg_loss: 0.26 -> 0.21 | Helped |
| Use ADF58 rather than KITTI fine-tuning | Scale-flow · DAVIS real-world | Pfg_loss: 13.62 -> 11.56 | Helped |
| Use ADF58 rather than KITTI fine-tuning | RAFT · DAVIS real-world | Sfg_loss: 0.29 -> 0.22 | Helped |
| Use ADF58 rather than KITTI fine-tuning | GMFlow · DAVIS real-world | Sfg_loss: 0.31 -> 0.26 | Helped |
| Train Scale-flow on ADF58 instead of KITTI raw/multi-frame data | Scale-flow · KITTI 2015 scene flow | Miderror: 288.99 -> 149.83 | Helped |
| Use more scenes, ADF58 instead of ADF15 (ablation) | RAFT · KITTI evaluation | Flepe | Helped |
| Add Mconf filtering (ablation) | RAFT trained on ADF15 · KITTI and Middlebury-A | Flepe | Helped |
| Add Mf motion foreground (ablation) | RAFT trained on ADF15 · KITTI | Flepe | Helped |
| Add Mf motion foreground (ablation) | RAFT trained on ADF15 · Middlebury-A | Flepe | Hurt |
| Add all masks Mconf, Mssim, Mdc, and Mf (ablation) | RAFT trained on ADF15 · KITTI 2015 | Flepe: 4.62 -> 4.19 | Helped |
| Add all masks Mconf, Mssim, Mdc, and Mf (ablation) | RAFT trained on ADF15 · KITTI 2012 | Flepe: 1.81 -> 1.62 | Helped |
| Replace synthetic training data with ADF15 | RAFT · KITTI evaluation | Flepe | Helped |
Surprising findings
- Self-supervised ADF-trained models generalize better to real-world DAVIS than KITTI-finetuned supervised models, despite using no real-world flow labels.
- Adding the motion foreground mask Mf improved KITTI but decreased Middlebury performance, as expected for static scenes.
Limitations
- Zip-NeRF cannot reconstruct reflective or transparent surfaces such as glass and water, so car and window regions are weak.
- Restricted to static scenes; dynamic foreground is only simulated with Bezier-curve slices.
- Road driving scenes were not captured for safety reasons, so KITTI performance lags behind daily-scene results.
Experimental setup
Datasets: ADF58 ADF15 KITTI 2012 KITTI 2015 Middlebury DAVIS Sintel FlyingThings3D FlyingChairs2 KITTI raw KITTI multi-view
Benchmarks: KITTI 2012 KITTI 2015 Middlebury DAVIS KITTI scene flow normalized evaluation
Metrics: end-to-end point error (EPE) outlier rate (Fall) SSIM loss photometric loss depth change rate error (Mid_error)
Baselines: MDFlow UPFlow UFlow SMMSF Scale-flow (KITTI-ft) RAFT (KITTI-ft) GMFlow Sintel FlyingThings3D FlyingChairs2
Builds on: Zip-NeRF RAFT Scale-flow instant-NGP
Closest papers by meaning
- Flow-NeRF: Joint Learning of Geometry, Poses, and Dense Flow within Unified Neural Representations (CVPR 2025 · similarity 0.87)
- ZeroFlow: Scalable Scene Flow via Distillation (ICLR 2024 · similarity 0.86)
- UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model (CVPR 2024 · similarity 0.86)
- Taming generative video models for zero-shot optical flow extraction (NeurIPS 2025 · similarity 0.86)
- Zero-Shot Monocular Scene Flow Estimation in the Wild (CVPR 2025 · similarity 0.85)
- Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.