Research map / Video / Video segmentation and tracking
Bridge Frame and Event: Common Spatiotemporal Fusion for High-Dynamic Scene Optical Flow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ComST-Flow, a frame-event optical flow method that uses a common spatiotemporal gradient space to locate boundary templates that guide dense (spatial) and continuous (temporal) correlation fusion for high-dynamic scenes.
Problem
Optical flow in high-dynamic scenes suffers spatial blur and temporal discontinuity from large displacement in frame imaging; directly fusing frame and event features is ineffective due to a large modality gap.
Approach
Builds a common spatiotemporal gradient space shared by frame and event data to localize reference boundary points, then uses these points to guide spatially dense frame correlation and temporally continuous event correlation fusion via a cross-modal transformer, with EKF tracking and K-Means clustering.
Key results
- On Fast-DSEC, ComST-Flow reaches EPE 0.58 and F1-all 1.96%, versus 0.71/2.38% for the best prior multimodal baseline shown in Table 1 (ABDA-Flow 0.87/2.89% for Fast-DSEC column ranking varies).
- On Event-KITTI, ComST-Flow achieves F1-all 3.83% and EPE 0.72, beating BFlow (4.36%) and ABDA-Flow (4.35%) in the table.
- On the proposed unseen day/night dataset, ComST-Flow obtains EPE 3.78 (day) and 3.95 (night), improving over ABDA-Flow at 3.97 and 3.97.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ComST-Flow instead of ABDA-Flow | Frame-event optical flow · Fast-DSEC | EPE: 0.71 -> 0.58 | Helped |
| add common fusion instead of direct fusion (ablation) | FlowFormer · Fast-DSEC | EPE: 0.84 -> 0.58 | Helped |
| add Lconsis_flow (ablation) | ComST-Flow · Fast-DSEC | EPE: 1.21 -> 0.93 | Helped |
| add Lkl alongside Lconsis_flow (ablation) | ComST-Flow · Fast-DSEC | EPE: 0.93 -> 0.89 | Helped |
| add LspaErr_corr alongside Lkl and Lconsis_flow (ablation) | ComST-Flow · Fast-DSEC | EPE: 0.89 -> 0.78 | Helped |
| add LtempErr_corr alongside Lkl and Lconsis_flow (ablation) | ComST-Flow · Fast-DSEC | EPE: 0.89 -> 0.65 | Helped |
| add both spatial and temporal correlation losses (ablation) | ComST-Flow · Fast-DSEC | EPE: 0.65 -> 0.58 | Helped |
| add correlation clustering (ablation) | ComST-Flow · Fast-DSEC | EPE: 0.64 -> 0.58 | Helped |
| use DBSCAN, GMM or K-Means clustering (ablation) | ComST-Flow · Fast-DSEC | EPE: 0.58; 0.59; 0.58 | No clear effect |
| add temporal tracking (ablation) | ComST-Flow · Fast-DSEC | EPE: 0.74 -> 0.58 | Helped |
| track correlation features rather than event signals (ablation) | ComST-Flow · Fast-DSEC | EPE: 0.71 -> 0.58 | Helped |
| track correlation features rather than visual features (ablation) | ComST-Flow · Fast-DSEC | EPE: 0.62 -> 0.58 | Helped |
Surprising findings
- Frame-only supervised RAFT/GMA and event-only E-RAFT show no obvious difference on slow motion, while event-based methods outperform frame-based on fast motion.
- Deblurring preprocessing improves frame-based flow only up to an upper limit, and flow backbone choice (RAFT vs FlowFormer) gives no obvious improvement compared with the fusion framework.
Limitations
- Fails for objects moving radially along the z-axis relative to the camera, which frame and event cameras cannot observe in x,y motion.
- Requires pixel-aligned coaxial frame-event hardware with beam splitter and careful calibration and time synchronization.
Experimental setup
Datasets: Event-KITTI DSEC KITTI2015
Benchmarks: Event-KITTI DSEC proposed frame-event dataset
Metrics: EPE F1-all TEPE
Baselines: ARFlow Selflow RAFT GMA E-RAFT BFlow ABDA-Flow
Builds on: RAFT FlowFormer ABDA-Flow BFlow E-RAFT
Closest papers by meaning
- Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow (CVPR 2024 · similarity 0.91)
- Injecting Frame-Event Complementary Fusion into Diffusion for Optical Flow in Challenging Scenes (NeurIPS 2025 · similarity 0.90)
- Motion Deblurring via Spatial-Temporal Collaboration of Frames and Events (AAAI 2024 · similarity 0.89)
- A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarking (NeurIPS 2025 · similarity 0.87)
- TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion (CVPR 2025 · similarity 0.86)
- StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.