Research map / Video / Video segmentation and tracking
Optical Flow for Spike Camera with Hierarchical Spatial-Temporal Spike Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiST-SFlow, a spike-camera optical flow network with a hierarchical spatial-temporal spike fusion module (InterF, IntraF, GTA) and a scene loss, trained on SPIFT and evaluated on PHM and real-captured spikes.
Problem
Optical flow estimation from binary spike streams of spike cameras, where Poisson and circuit noise make spike occurrences random and fluctuating, so conventional feature matching is unreliable.
Approach
Replace single-shot temporal fusion of spikes with a hierarchical scheme that alternates inter-moment 3D-conv temporal/spatial fusion with intra-moment residual filtering, aggregates features across levels into a spike representation, and supervises it with an auxiliary scene-brightness loss.
Key results
- On PHM at delta-t=10, HiST-SFlow reaches average AEPE 0.640 and PO% 15.54, versus 0.813 AEPE for the best competing RAFT-style baseline (SCFlow 0.863).
- On PHM at delta-t=20, HiST-SFlow reaches average AEPE 1.417 and PO% 19.73, improving on KPA-Flow's 1.649 AEPE.
- Plugging HiST into GMA, KPA-Flow and GMFlowNet reduces AEPE by roughly 20-30% on the PHM dataset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HiST-SFlow instead of RAFT | spike-based optical flow · PHM Δt = 10 | AEPE: 0.813 -> 0.640 | Helped |
| use HiST-SFlow instead of CRAFT | spike-based optical flow · PHM Δt = 10 | PO%: 23.94 -> 15.54 | Helped |
| use HiST-SFlow instead of RAFT | spike-based optical flow · PHM Δt = 20 | AEPE: 1.628 -> 1.417 | Helped |
| use HiST-SFlow instead of RAFT | spike-based optical flow · PHM Δt = 20 | PO%: 28.94 -> 19.73 | Helped |
| add InterF, IntraF, and scene loss (ablation) | HiST-SFlow · PHM Δt = 10 | AEPE: 0.986 -> 0.640 | Helped |
| add InterF, IntraF, and scene loss (ablation) | HiST-SFlow · PHM Δt = 20 | AEPE: 2.095 -> 1.417 | Helped |
| add IntraF (ablation) | HiST-SFlow with InterF and scene loss · PHM Δt = 20 | PO%: 21.40 -> 19.73 | Helped |
| add IntraF (ablation) | HiST-SFlow with InterF and scene loss · PHM Δt = 10 | AEPE: 0.675 -> 0.640 | No clear effect |
| add scene loss (ablation) | HiST-SFlow with InterF and IntraF · PHM Δt = 10 | AEPE: 0.676 -> 0.640 | Helped |
| add scene loss (ablation) | HiST-SFlow with InterF and IntraF · PHM Δt = 20 | PO%: 22.79 -> 19.73 | Helped |
| replace window-based representation with HiST (ablation) | spike-based optical flow · PHM Δt = 10 | AEPE: 0.868 -> 0.640 | Helped |
| replace interval-based representation with HiST (ablation) | spike-based optical flow · PHM Δt = 20 | AEPE: 1.824 -> 1.417 | Helped |
Surprising findings
- Straightforwardly adapted image-based optical flow architectures trained on spikes (e.g., GMFlow, Flow1D) can underperform the spike-specific SCFlow on some tracks.
Limitations
- Evaluation is limited to simulated PHM and a small set of real-captured scenes.
- Scene loss relies on ground-truth brightness available only in the simulated SPIFT dataset.
Experimental setup
Datasets: SPIFT PHM
Benchmarks: PHM real-captured spike data
Metrics: AEPE PO%
Baselines: SCFlow RAFT GMA Flow1D KPA-Flow GMFlow GMFlowNet CRAFT FlowFormer
Builds on: RAFT CRAFT SCFlow
Closest papers by meaning
- Boosting Spike Camera Image Reconstruction from a Perspective of Dealing with Spike Fluctuations (CVPR 2024 · similarity 0.90)
- Rethinking High-speed Image Reconstruction Framework with Spike Camera (AAAI 2025 · similarity 0.87)
- Spike-guided Motion Deblurring with Unknown Modal Spatiotemporal Alignment (CVPR 2024 · similarity 0.86)
- Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras (NeurIPS 2024 · similarity 0.86)
- Recognizing Ultra-High-Speed Moving Objects with Bio-Inspired Spike Camera (AAAI 2024 · similarity 0.86)
- Finding Visual Saliency in Continuous Spike Stream (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.