Research map / 3D vision / Depth and stereo
Shape and Texture: What Influences Reliable Optical Flow Estimation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Flow-R, a KITTI-derived robustness test set that edits object texture, shape, or adds unseen objects, and finds SOTA optical flow models are more sensitive to shape changes and tend to ignore unseen objects.
Problem
Optical flow methods are evaluated mainly for benchmark accuracy, while their robustness to object shape/texture changes and unseen objects, important for safety-critical uses like autonomous driving, is little studied.
Approach
Robustness evaluation by semantically editing real benchmark images (diffusion-based texture replacement, object rescaling with flow update, and pasting unseen real objects) rather than adding adversarial perturbations, then analyzing failure modes via feature and decoder-iteration visualization.
Key results
- Across SOTA methods, errors rise notably on Flow-R-S (shape scaling; e.g., RAFT Fl-all 1.5 on KITTI-style fine-tuned setting rising to 10.2) while texture replacement (Flow-R-T) changes results little.
- Adding unseen real-world objects markedly increases error: RAFT AEPE foreground 20.4 for unseen vs 7.43 foreground... (add-unseen all AEPE 25.68 vs add-seen 11.26 for RAFT's all-pixel metric), indicating methods ignore unseen objects.
- Feature and GRU-iteration analyses show encoder retains unseen-object features, but the GRU decoder gradually suppresses them over iterations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace object texture while keeping its shape and motion | SOTA optical flow methods · Flow-R-T | AEPE: 5.18 vs. 5.97 | No clear effect |
| Replace object shape by scaling it | SOTA optical flow methods · Flow-R-S | AEPE: 5.18 vs. 5.97 | Hurt |
| Replace texture in the first frame only | FlowFormer · Texture attack | AEPE: 0.64 | No clear effect |
| Scale the object in the first frame only | FlowFormer · Shape attack | AEPE: 2.04 | Hurt |
| Add a seen object from KITTI | SOTA optical flow methods · Add Seen | AEPE: 2.43 | No clear effect |
| Add an unseen object | SOTA optical flow methods · Add Unseen | AEPE: 7.43 | Hurt |
| Add an unseen object | SOTA optical flow methods · Add Unseen foreground | AEPE: 20.53 | Hurt |
| Add an unseen object | SOTA optical flow methods · Add Unseen background | AEPE: 7.18 | Hurt |
| Use iterative decoder refinement on inputs containing an unseen object | GMA · Unseen-object examples | SSIM | Mixed |
| Process unseen-object features through the GRU block across iterations | GMA and FlowFormer++ · Unseen-object examples | feature similarity | Hurt |
Surprising findings
- Texture changes barely affect flow accuracy, even when only one frame's texture is altered, suggesting matching relies less on texture than shape.
- Unseen objects are ignored even when encoder features clearly represent them, with the decoder GRU suppressing them across iterations.
Limitations
- Flow-R relies on manual quality control of SAM masks and generated images, limiting scale.
- Shape reduction (theta < 1) requires inpainting and masking of background flow, adding confounds.
- Evaluation is black-box on a limited set of methods and KITTI-based scenes.
Experimental setup
Datasets: KITTI 2015 Flow-R (constructed from KITTI) Chairs FlyingThings3D Sintel HD1K
Benchmarks: KITTI 2015 Flow-R-T Flow-R-S Flow-R-O
Metrics: AEPE Fl-all
Baselines: RAFT GMA GMFlow FlowFormer FlowFormer++ SAMFlow PWC-Net
Builds on: RAFT ControlNet Segment Anything Model KITTI
Closest papers by meaning
- DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework (CVPR 2025 · similarity 0.86)
- CEDFlow: Latent Contour Enhancement for Dark Optical Flow Estimation (AAAI 2024 · similarity 0.85)
- SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model (AAAI 2024 · similarity 0.85)
- UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model (CVPR 2024 · similarity 0.85)
- HuPerFlow: A Comprehensive Benchmark for Human vs. Machine Motion Estimation Comparison (CVPR 2025 · similarity 0.84)
- DifFlow3D: Toward Robust Uncertainty-Aware Scene Flow Estimation with Iterative Diffusion-Based Refinement (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.