Research map / Video / Video segmentation and tracking
Spike-guided Motion Deblurring with Unknown Modal Spatiotemporal Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UaSDN, a three-stage network that uses spike-camera streams to guide motion deblurring of RGB images without requiring known spatial or temporal alignment between the two sensors.
Problem
Motion deblurring of RGB frames guided by spike camera streams when the spatial and temporal alignment between the spike and RGB cameras is unknown, as happens in real-world setups with vibrations and sync errors.
Approach
A three-stage pipeline: NAFNet-based basic deblurring, then bi-directional deformable alignment (BiDMA) between image and spike features, then flow-guided multi-scale dense fusion that warps coarse spike-derived grayscale estimates using optical flow.
Key results
- UaSDN reaches 35.78 dB PSNR / 0.964 SSIM on X4K1000FPS with unaligned spikes, versus 34.47 dB for SpkDeblurNet and 34.13 dB for NAFNet.
- On REDS, UaSDN achieves 33.61 dB PSNR, outperforming EFNet (31.74 dB) and HINet (31.10 dB).
- With precise spatiotemporal alignment, UaSDN reaches 36.92 dB on X4K1000FPS, exceeding EFNet and REFID by 0.56 and 0.62 dB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use UaSDN instead of SpkDeblurNet | spike-guided motion deblurring · X4K1000FPS | PSNR: 34.47 -> 35.78 | Helped |
| use UaSDN instead of EFNet | spike-guided motion deblurring · REDS | PSNR: 31.74 -> 33.61 | Helped |
| add Stage 2 bi-directional deformable alignment to Stage 1 (ablation) | UaSDN · X4K1000FPS | PSNR: 32.30 -> 32.90 | Helped |
| add Stage 2 bi-directional deformable alignment to Stage 1 (ablation) | UaSDN · X4K1000FPS | SSIM: 0.914 -> 0.923 | Helped |
| add Stage 3 flow-guided dense fusion to Stage 1 + Stage 2 (ablation) | UaSDN · X4K1000FPS | PSNR: 32.90 -> 35.78 | Helped |
| add Stage 3 flow-guided dense fusion to Stage 1 + Stage 2 (ablation) | UaSDN · X4K1000FPS | SSIM: 0.923 -> 0.964 | Helped |
Surprising findings
- Image-only NAFNet underperforms the spike-guided EFNet on REDS (30.73 vs 31.74 dB), while EFNet is below HINet on X4K1000FPS.
Limitations
- Evaluated on simulated spikes generated from high-frame-rate videos; real-world evaluation is only qualitative on a small custom setup.
- Performance drops when sampled spikes are not within the image exposure window at all.
- Relies on a pretrained optical flow network and a three-stage separately trained pipeline.
Experimental setup
Datasets: X4K1000FPS REDS
Benchmarks: X4K1000FPS REDS
Metrics: PSNR SSIM
Baselines: HINet NAFNet EFNet REFID SpkDeblurNet
Builds on: NAFNet deformable convolution EMA-VFI RAFT-style optical flow estimation
Closest papers by meaning
- SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams (NeurIPS 2024 · similarity 0.91)
- DeblurSR: Event-Based Motion Deblurring under the Spiking Representation (AAAI 2024 · similarity 0.89)
- Rethinking High-speed Image Reconstruction Framework with Spike Camera (AAAI 2025 · similarity 0.88)
- Spk2SRImgNet: Super-Resolve Dynamic Scene from Spike Stream via Motion Aligned Collaborative Filtering (CVPR 2025 · similarity 0.88)
- Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras (NeurIPS 2024 · similarity 0.87)
- USP-Gaussian: Unifying Spike-based Image Reconstruction, Pose Correction and Gaussian Splatting (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.