Research map / Video / Video segmentation and tracking
Injecting Frame-Event Complementary Fusion into Diffusion for Optical Flow in Challenging Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Diff-ABFlow, a diffusion-based optical flow framework that fuses frame and event features via attention-guided appearance-boundary fusion and denoises flow with a multi-condition GRU decoder, targeting high-speed and low-light scenes.
Problem
Optical flow estimation degrades in high-speed (motion blur) and low-light (insufficient illumination) scenes, where frame features lose texture and boundary completeness and existing discriminative/generative decoders are sensitive to degraded visual features.
Approach
Fuse frame appearance features with event boundary features via attention-guided appearance-boundary fusion, and reformulate flow estimation as a conditional denoising process (DDIM-style with GRU memory-slot decoder) conditioned on fused visual, motion, and time features.
Key results
- On HS-KITTI, Diff-ABFlow achieves EPE 0.46 and Fl-all 1.12, versus 0.55/1.90 for BFlow (frame-event).
- On unseen HS-FEFD, Diff-ABFlow reaches EPE 4.69 and Fl-all 28.77, outperforming FlowDiffuser (6.72 / 49.51).
- Ablation shows Attention-ABF plus TVM-MCA plus memory slot reduces Fl-all from 4.73 (baseline) to 1.12 on the HS-KITTI setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use frame-event input instead of frame-only input (ablation) | Diff-ABFlow · HS-KITTI | EPE: 0.58 -> 0.46 | Helped |
| Use frame-event input instead of event-only input (ablation) | Diff-ABFlow · HS-KITTI | EPE: 0.65 -> 0.46 | Helped |
| Replace concatenation fusion with attention-guided fusion (ablation) | Diff-ABFlow · HS-KITTI | EPE: 0.57 -> 0.46 | Helped |
| Replace weighted fusion with attention-guided fusion (ablation) | Diff-ABFlow · HS-KITTI | EPE: 0.55 -> 0.46 | Helped |
| Replace FlowFormer with diffusion models (ablation) | Optical flow backbone · HS-KITTI | EPE: 0.59 -> 0.46 | Helped |
| Replace GAN with diffusion models (ablation) | Optical flow backbone · HS-KITTI | EPE: 0.65 -> 0.46 | Helped |
| Use MGDD instead of U-Net (ablation) | Diff-ABFlow diffusion module · HS-KITTI | EPE: 0.63 -> 0.46 | Helped |
| Use MGDD instead of Conditional-RDD (ablation) | Diff-ABFlow diffusion module · HS-KITTI | EPE: 0.57 -> 0.46 | Helped |
| Increase denoising steps from K=1 to K=4 (ablation) | Diff-ABFlow · HS-KITTI | EPE: 0.61 -> 0.46 | Helped |
| Increase denoising steps from K=4 to K=5 (ablation) | Diff-ABFlow · HS-KITTI | EPE: 0.46 -> 0.46 | No clear effect |
| Add Attention-ABF to the model (ablation) | Diff-ABFlow · HS-KITTI | EPE: 0.59 -> 0.46 | Helped |
| Add TVM-MCA to the model (ablation) | Diff-ABFlow · HS-KITTI | EPE: 0.73 -> 0.54 | Helped |
Surprising findings
- Visual enhancement (deblurring, low-light enhancement) often fails to improve and sometimes worsens optical flow metrics.
Limitations
- Fails on textureless planes since neither frame nor event captures discriminative signals there.
- Inference time (~198 ms) is high for real-time use.
- Proposed HS-FEFD and LL-FEFD datasets are used only for generalization evaluation, not training.
Experimental setup
Datasets: KITTI2015 DSEC AutoFlow FlyingChairs FlyingThings MPI-Sintel HS-KITTI LL-KITTI HS-DSEC LL-DSEC
Benchmarks: HS-KITTI LL-KITTI HS-DSEC LL-DSEC HS-FEFD LL-FEFD
Metrics: EPE Fl-all inference time parameters memory consumption
Baselines: GMA FlowFormer++ E-RAFT BFlow ABDA-Flow FlowDiffuser
Builds on: DDPM DDIM RAFT-style iterative refinement FlowDiffuser BFlow
Closest papers by meaning
- Bridge Frame and Event: Common Spatiotemporal Fusion for High-Dynamic Scene Optical Flow (CVPR 2025 · similarity 0.90)
- DifFlow3D: Toward Robust Uncertainty-Aware Scene Flow Estimation with Iterative Diffusion-Based Refinement (CVPR 2024 · similarity 0.88)
- Hierarchical Flow Diffusion for Efficient Frame Interpolation (CVPR 2025 · similarity 0.87)
- EDCFlow: Exploring Temporally Dense Difference Maps for Event-based Optical Flow Estimation (CVPR 2025 · similarity 0.86)
- DiffSF: Diffusion Models for Scene Flow Estimation (NeurIPS 2024 · similarity 0.86)
- Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.