Research map / Video / Video segmentation and tracking
Semi-Supervised State-Space Model with Dynamic Stacking Filter for Real-World Video Deraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VDMamba, a dual-branch spatio-temporal state-space model for video deraining with a differentiable dynamic stacking filter and a median-stacking semi-supervised loss, plus a new real-world rainy detection/tracking benchmark (RVDT).
Problem
Video deraining models trained on paired synthetic data generalize poorly to real rain, and the benefit of deraining for downstream detection and tracking in rainy videos is under-explored.
Approach
A dual-branch spatial/temporal Mamba-style state-space architecture with a dynamic stacking filter that differentiably approximates mean, median, min and max pixel fusion across aligned frames, and a median-stacking pseudo-label loss exploiting rain sparsity for unpaired real videos.
Key results
- On RainSynLight25 the method reaches 37.53 dB PSNR / 0.9812 SSIM, above the next best MFGAN at 36.99 dB / 0.9760.
- On RVDT, detection mAP1 improves from 39.73 (rainy) to 42.51 and tracking MOTA from 28.0 to 31.8 with the proposed deraining.
- Ablation: removing the dynamic stacking filter drops PSNR from 39.74 to 39.21 on NTURain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VDMamba instead of the strongest listed baseline | Video deraining · RainSynLight25 | PSNR: 36.99 -> 37.53 | Helped |
| use VDMamba instead of the strongest listed baseline | Video deraining · RainSynComplex25 | PSNR: 32.70 -> 32.89 | Helped |
| use VDMamba instead of the strongest listed baseline | Video deraining · NTURain | PSNR: 38.92 -> 39.74 | Helped |
| use VDMamba instead of the strongest listed baseline | Rainy-scene detection and tracking · RVDT | mAP1: 41.59 -> 42.51 | Helped |
| use VDMamba instead of the strongest listed baseline | Rainy-scene detection and tracking · RVDT | mAP2: 44.22 -> 46.27 | Helped |
| replace S3ML and TSML with residual blocks (ablation) | VDMamba · NTURain | PSNR: 39.74 -> 38.54 | Hurt |
| replace S3ML with residual blocks (ablation) | VDMamba · NTURain | PSNR: 39.74 -> 39.02 | Hurt |
| replace TSML with residual blocks (ablation) | VDMamba · NTURain | PSNR: 39.74 -> 39.24 | Hurt |
| use direct pre-trained flow estimation for multiple frame pairs (ablation) | VDMamba optical-flow pipeline · NTURain | PSNR: 39.74 -> 39.65 | Hurt |
| transfer optical flow on rainy frames (ablation) | VDMamba optical-flow pipeline · NTURain | PSNR: 39.74 -> 39.03 | Hurt |
| transfer optical flow on clean frames (ablation) | VDMamba optical-flow pipeline · NTURain | PSNR: 39.74 -> 38.51 | Hurt |
| remove the dynamic stacking filter (ablation) | VDMamba · NTURain | PSNR: 39.74 -> 39.21 | Hurt |
Limitations
- Relies on pre-trained optical flow estimator (LiteFlowNet3) and pseudo-labels whose quality depends on rain sparsity and alignment.
- Stacking loss requires a threshold (theta) and slack (delta) that are hand-set.
Experimental setup
Datasets: RainSynLight25 RainSynComplex25 NTURain RVDT
Benchmarks: RainSynLight25 RainSynComplex25 NTURain RVDT
Metrics: PSNR SSIM mAP IDF1 MOTA MOTP
Baselines: FastDeRain SpacCNN J4RNet DRSformer ESTINet MFGAN RainMamba SLDNet S2VD
Builds on: Mamba LiteFlowNet3 RainMamba DRSformer GTR
Closest papers by meaning
- Semi-Supervised Video Deraining With Dynamical Rain Generator (CVPR 2021 · similarity 0.90)
- Memory Oriented Transfer Learning for Semi-Supervised Image Deraining (CVPR 2021 · similarity 0.89)
- Harnessing Joint Rain-/Detail-aware Representations to Eliminate Intricate Rains (ICLR 2024 · similarity 0.88)
- Towards Universal Rainy Image Restoration: Benchmark and Baseline (AAAI 2025 · similarity 0.87)
- Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining (CVPR 2024 · similarity 0.86)
- Boosting Image De-Raining via Central-Surrounding Synergistic Convolution (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.