Research map / Video / Video segmentation and tracking
Learning Phase Distortion with Selective State Space Models for Video Turbulence Mitigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MambaTM, a Mamba-based video turbulence mitigation network guided by a learned latent phase distortion (LPD) representation from a conditional VAE that replaces the Zernike-based simulator.
Problem
Video turbulence mitigation needs large spatiotemporal receptive fields and degradation awareness, but convolutional and self-attention methods are slow or memory-hungry, and Zernike-based degradation estimation is ill-posed and slow.
Approach
Replace the Zernike-based turbulence simulator with a learned latent phase distortion (LPD) representation via a conditional VAE/ReBlurNet, and use the LPD to guide the selective state space (Mamba) blocks' input-dependent parameters, giving a linear-complexity global receptive field across space and time.
Key results
- On ATSyn-dynamic, MambaTM reaches 29.9151 dB PSNR overall vs 29.4222 for DATUM.
- On the TMT dynamic dataset, MambaTM achieves 28.9049 PSNR at 55.4 FPS versus 32.7 FPS for DATUM (512x512 on A100).
- On Turb-Text, MambaTM reaches 97.80 / 99.35 / 98.15 text recognition rates (CRNN/DAN/ASTER), above DATUM's 93.55 / 97.95 / 97.25.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Zernike coefficients with learned LPD representation | MambaTM re-degradation · Re-degradation test | PSNR: 25.84 / 31.17 -> 34.08 | Helped |
| replace Zernike-based simulator with LPD simulator | turbulence simulation · Re-degradation simulator | speed (s): 0.16 ∼ 6.10 -> 0.02 | Helped |
| use MambaTM instead of DATUM | video turbulence mitigation · ATSyn-dynamic Overall | PSNR: 29.4222 -> 29.9151 | Helped |
| use MambaTM instead of DATUM | video turbulence mitigation · TMT synthetic dynamic dataset | FPS: 32.7 -> 55.4 | Helped |
| use MambaTM instead of DATUM | video turbulence mitigation · ATSyn-static | PSNR: 26.7623 -> 27.0082 | Helped |
| add LPD-guided Mamba blocks (ablation) | MambaTM · ATSyn-dynamic ablation | PSNR: 29.6679 -> 29.9151 | Helped |
| add LPD re-degradation supervision (ablation) | MambaTM · ATSyn-dynamic ablation | PSNR: 29.1274 -> 29.7495 | Helped |
| remove local Hilbert scan (ablation) | MambaTM · ATSyn-dynamic ablation | PSNR: 29.9151 -> 29.4933 | Hurt |
| remove time-first scan (ablation) | MambaTM · ATSyn-dynamic ablation | PSNR: 29.9151 -> 29.6077 | Hurt |
| remove space-first scan (ablation) | MambaTM · ATSyn-dynamic ablation | PSNR: 29.9151 -> 29.5058 | Hurt |
| add multi-scale modulation to ReBlurNet (ablation) | ReBlurNet · Re-degradation network ablation | PSNR: +1 dB | Helped |
| replace PlainUNet with NAFNet (ablation) | ReBlurNet · Re-degradation network ablation | PSNR: 43.26 -> 46.72 | Helped |
Surprising findings
- Supervising Zernike coefficients directly failed to converge in re-degradation training, while image-level supervision with LPD worked.
Limitations
- Two separate models are needed for dynamic and static scenes.
- Real-world evaluation relies on downstream text recognition and qualitative comparison since no ground truth exists.
Experimental setup
Datasets: ATSyn LSDIR TMT synthetic dataset Turb-Text BRIAR
Benchmarks: ATSyn-dynamic TMT dynamic synthetic dataset ATSyn-static Turb-Text
Metrics: PSNR SSIM LPIPS FPS text recognition rate
Baselines: VRT RVRT ESTRNN RNN-MBP TSRWGAN TMT DATUM PiRN Turb-Seg-Res
Builds on: Mamba Zernike-based turbulence simulation (P2S) DATUM NAFNet VRT
Closest papers by meaning
- Spatio-Temporal Turbulence Mitigation: A Translational Perspective (CVPR 2024 · similarity 0.91)
- Temporally Consistent Atmospheric Turbulence Mitigation with Neural Representations (NeurIPS 2024 · similarity 0.88)
- NB-GTR: Narrow-Band Guided Turbulence Removal (CVPR 2024 · similarity 0.84)
- Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence (CVPR 2024 · similarity 0.84)
- PlaNet: Learning to Mitigate Atmospheric Turbulence in Planetary Images (AAAI 2025 · similarity 0.84)
- Diffeomorphic Template Registration for Atmospheric Turbulence Mitigation (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.