Research map / Video / Video segmentation and tracking
Temporally Consistent Atmospheric Turbulence Mitigation with Neural Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConVRT, a self-supervised neural video representation with a temporal deformation field and spatial content field, regularized along time, to make any video atmospheric turbulence mitigation method temporally consistent.
Problem
Video atmospheric turbulence mitigation (ATM) methods produce temporally inconsistent, flickering restorations because turbulence distortions vary stochastically across space and time.
Approach
Decouple video into a spatial content field and a temporal deformation field, and regularize the temporal feature map (reduced temporal resolution, smaller deformation MLP, disparity-weighted warp loss) so the network low-pass filters turbulence-induced temporal variations; the representation is overfit per clip to outputs of any base ATM method.
Key results
- Applied on top of DATUM, ConVRT reduces average warp error on HeatChamber from 22.43 to 17.77 and improves PSNR from 19.25 to 19.46.
- On a synthetic dataset with TurbNet as base, the full ConVRT configuration (Tres=5 with temporal loss) raises PSNR from 22.57 to 24.31 and SSIM from 0.673 to 0.709.
- Table 2 shows temporal consistency gains (warp error, flow TV, slice TV) across TMT, DATUM, TurbNet, VRT, and standalone use on the real-world datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ConVRT to DATUM | DATUM · HeatChamber | Ewarp: 22.43 -> 17.77 | Helped |
| Apply ConVRT to DATUM | DATUM · HeatChamber | PSNR: 19.25 -> 19.46 | Helped |
| Apply ConVRT to DATUM | DATUM · HeatChamber | Flowtv: 5794.12 -> 2509.21 | Helped |
| Apply ConVRT to DATUM | DATUM · OTIS | Flowtv: 6742.56 -> 871.26 | Helped |
| Apply ConVRT to DATUM | DATUM · CLEAR | Slicetv: 123.32 -> 124.93 | Hurt |
| Apply ConVRT to DATUM | DATUM · TSRWGAN | Flowtv: 2279.42 -> 411.80 | Helped |
| Set ConVRT temporal resolution to 15 (ablation) | ConVRT on TurbNet · Synthetic turbulence | PSNRImg: 22.57 -> 23.29 | Helped |
| Set ConVRT temporal resolution to 8 (ablation) | ConVRT on TurbNet · Synthetic turbulence | PSNRImg: 22.57 -> 23.91 | Helped |
| Set ConVRT temporal resolution to 5 (ablation) | ConVRT on TurbNet · Synthetic turbulence | PSNRImg: 22.57 -> 24.16 | Helped |
| Add Ltemp with Tres=5 (ablation) | ConVRT on TurbNet · Synthetic turbulence | PSNRImg: 24.16 -> 24.31 | Helped |
Surprising findings
- Applied standalone to original frames without any base ATM method, ConVRT still improves temporal consistency (warp error on HeatChamber drops from 24.33 to 16.19).
Limitations
- Performance depends on accurate video representation and is optimized for short clips; longer sequences and complex motions are not handled.
- Runtime (about 10 minutes per 25-frame clip) remains costly for larger or more complex sequences.
- No error bars or statistical significance reported.
Experimental setup
Datasets: HeatChamber OTIS BVI-CLEAR TSR-WGAN dataset DOST
Benchmarks: HeatChamber OTIS BVI-CLEAR (CLEAR) TSR-WGAN dataset
Metrics: PSNR SSIM warp error (Ewarp) Flow TV Slice TV PSNR x-t
Baselines: TMT DATUM TurbNet VRT Mao et al. Li et al.
Builds on: VRT TMT DATUM TurbNet MiDaS
Closest papers by meaning
- Spatio-Temporal Turbulence Mitigation: A Translational Perspective (CVPR 2024 · similarity 0.89)
- Learning Phase Distortion with Selective State Space Models for Video Turbulence Mitigation (CVPR 2025 · similarity 0.88)
- Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence (CVPR 2024 · similarity 0.85)
- Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal (CVPR 2024 · similarity 0.84)
- Diffeomorphic Template Registration for Atmospheric Turbulence Mitigation (CVPR 2024 · similarity 0.83)
- End-to-End Video Semantic Segmentation in Adverse Weather using Fusion Blocks and Temporal-Spatial Teacher-Student Learning (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.