Research map / Video / Video segmentation and tracking
Spatio-Temporal Turbulence Mitigation: A Translational Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DATUM, a recurrent deep video network for atmospheric turbulence mitigation that embeds classical TM steps (registration, lucky fusion, deblurring) as inductive biases, plus the ATSyn physics-based synthetic dataset.
Problem
Turbulence mitigation from image sequences is slow and generalizes poorly to real dynamic scenes; existing deep video TM networks lack turbulence-specific inductive biases and synthetic training data is often unrealistic or slow to generate.
Approach
Encodes classical TM insights into a recurrent network: deformable attention alignment of features to a recurrently updated tilt-free reference (DAAB + GRUM), temporal-channel self-attention for lucky fusion (MTCSA), and a twin decoder that rectifies tilt before deblurring; trained on ATSyn, a Zernike-based physics simulator dataset.
Key results
- On ATSyn-dynamic, DATUM reaches 29.42 dB PSNR overall vs 28.77 for RVRT, at 9.17 FPS versus RVRT's 2.43 FPS.
- On the TMT synthetic dynamic set, DATUM achieves 28.59 dB PSNR vs 27.74 for TMT, with about 5x fewer parameters and over 10x faster inference.
- On the turbulence text dataset, DATUM achieves 93.55/97.95/97.25% CRNN/DAN/ASTER recognition accuracy, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DATUM instead of RVRT | video turbulence mitigation on TMT synthetic dynamic scene · TMT synthetic dynamic scene | PSNR: 27.8512 -> 28.5875 | Helped |
| Use DATUM instead of RVRT | video turbulence mitigation on ATSyn-dynamic · ATSyn-dynamic overall | PSNR: 28.7672 -> 29.4222 | Helped |
| Use DATUM instead of ESTRNN | static-scene turbulence mitigation · ATSyn-static | PSNR: 26.23 -> 26.76 | Helped |
| Add GRUM (ablation) | DATUM · ATSyn-dynamic | PSNR: 28.87 -> 29.06 | Helped |
| Add DAAB (ablation) | DATUM with GRUM · ATSyn-dynamic | PSNR: 29.06 -> 29.33 | Helped |
| Add twin decoder (ablation) | DATUM with GRUM and DAAB · ATSyn-dynamic | PSNR: 29.33 -> 29.42 | Helped |
| Increase MTCSA window from 1 frame to 3 frames (ablation) | DATUM base · ATSyn-dynamic | PSNR: 28.62 -> 28.79 | Helped |
| Increase MTCSA window from 3 frames to 5 frames (ablation) | DATUM base · ATSyn-dynamic | PSNR: 28.79 -> 28.87 | Helped |
| Increase MTCSA window from 5 frames to 7 frames (ablation) | DATUM base · ATSyn-dynamic | PSNR: 28.87 -> 28.92 | Helped |
| Use more input frames (ablation) | DATUM · ATSyn-static | PSNR | Helped |
| Train with ATSyn-dynamic rather than the simulator in [72] (ablation) | DATUM on BRIAR face retrieval · BRIAR face retrieval | Rank 5: 38.83% -> 39.18% | Helped |
Surprising findings
- Models trained on ATSyn generalize better to real turbulence than the same architectures trained on the TMT dataset or their original checkpoints.
Limitations
- Temporal fusion window and frame count trade off performance against quadratic cost of MTCSA
- Real-world evaluation relies on downstream recognition/face-recognition proxies because no ground truth exists
Experimental setup
Datasets: ATSyn-dynamic ATSyn-static TMT synthetic dynamic dataset Places BRIAR Turbulence text dataset (UG2+)
Benchmarks: ATSyn-dynamic ATSyn-static TMT synthetic dynamic dataset Turbulence text dataset BRIAR face recognition
Metrics: PSNR SSIM (complex wavelet) FPS model size text recognition accuracy face rank-k retrieval accuracy
Baselines: TMT TSRWGAN TurbNet VRT RVRT ESTRNN RNN-MBP NDIR AT-DDPM PiRN
Builds on: Zernike-based turbulence simulator guided deformation attention RVRT SPyNet
Closest papers by meaning
- Learning Phase Distortion with Selective State Space Models for Video Turbulence Mitigation (CVPR 2025 · similarity 0.91)
- Temporally Consistent Atmospheric Turbulence Mitigation with Neural Representations (NeurIPS 2024 · similarity 0.89)
- PlaNet: Learning to Mitigate Atmospheric Turbulence in Planetary Images (AAAI 2025 · similarity 0.86)
- NB-GTR: Narrow-Band Guided Turbulence Removal (CVPR 2024 · similarity 0.85)
- Diffeomorphic Template Registration for Atmospheric Turbulence Mitigation (CVPR 2024 · similarity 0.85)
- Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.