Research map / Video / Video segmentation and tracking
Learning Truncated Causal History Model for Video Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Turtle, a causal video restoration model that summarizes a truncated history of frame latents with similarity-based top-k motion-compensated retrieval and a frame history router, achieving state-of-the-art on multiple video restoration tasks at lower compute.
Problem
Video restoration requires modeling motion-driven transition dynamics across frames, but parallel multi-frame methods are memory-hungry and recurrent methods accumulate errors and cannot parallelize well.
Approach
A U-Net style causal model whose decoder conditions each frame on a truncated history of previously restored frame features via a State Align Block (top-k patch attention for implicit motion compensation) and a Frame History Router (cross-frame channel attention), with a historyless encoder and clip-based parallel training.
Key results
- Night video deraining PSNR 29.26 dB vs. 26.73 for NightRain (+2.53 dB).
- Video desnowing PSNR 26.02 dB vs. 25.06 for SVDNet; real-world deblurring 33.58 dB vs. 31.58 for CDVD-TSP.
- Video SR 4x PSNR 25.30 dB vs. 23.94 for EAVSR+; 181 MACs (G) at 256x256 vs. 1182 for RVRT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Turtle's causal history restoration method | Video restoration · SynNightRain test | PSNR: 29.26 dB; +2.53 dB over NightRain | Helped |
| Use Turtle's causal history restoration method | Video restoration · RVSD test | PSNR: 26.02 dB; +0.96 dB | Helped |
| Use Turtle's causal history restoration method | Video restoration · BSD 3ms-24ms | PSNR: 33.58 dB; +2.0 dB | Helped |
| Use Turtle's causal history restoration method | Video restoration · GoPro | PSNR: 34.50 dB; +0.34 dB | Helped |
| Use Turtle's causal history restoration method | Video restoration · VRDS | PSNR: 32.01 dB; +0.99 dB | Helped |
| Use Turtle's causal history restoration method | Video restoration · MVSR 4× | PSNR: 25.30 dB; +1.36 dB | Helped |
| Use Turtle's blind video denoising method | Blind video denoising · DAVIS σ=30 and σ=50 | PSNR: average drop of -0.3 dB to BSVD-64 | Hurt |
| Use Turtle's blind video denoising method | Blind video denoising · Set8 σ=30 and σ=50 | PSNR: +0.31 dB at σ=30; +0.34 dB at σ=50 | Helped |
| Add CHM to the no-CHM baseline (ablation) | Turtle on GoPro (small model) · GoPro | PSNR: 31.84 -> 32.26 | Helped |
| Add the State Align Block (φ) (ablation) | Turtle on GoPro (small model) · GoPro | PSNR: 32.07 -> 32.26 | Helped |
| Increase truncation factor from τ=3 to τ=5 (ablation) | Turtle on GoPro (small model) · GoPro | PSNR: 32.26 -> 32.26 | No clear effect |
| Use τ=1 instead of the full method's τ=3 (ablation) | Turtle on GoPro (small model) · GoPro | PSNR: 32.26 -> 32.15 | Hurt |
Surprising findings
- Increasing the truncation factor beyond three frames gave no PSNR improvement on GoPro ablations.
- On DAVIS blind video denoising, Turtle averages about 0.3 dB below BSVD-64 despite winning on Set8.
Limitations
- Spurious patch correspondences (e.g., repetitive zebra stripes) and severe degradation can hurt history retrieval.
- Gains over strong baselines are task-dependent; underperforms BSVD-64 on DAVIS denoising.
Experimental setup
Datasets: SynNightRain RSVD GoPro BSD 3ms-24ms VRDS MVSR4x DAVIS
Benchmarks: SynNightRain RSVD BSD 3ms-24ms GoPro VRDS MVSR4x DAVIS Set8
Metrics: PSNR SSIM MACs
Baselines: FDM NightRain MetaRain SVDNet RDDNet BasicVSR++ RVRT VRT DSTNet ViMPNet BSVD-64 EAVSR+
Builds on: Transformer U-Net state space models BasicVSR++
Closest papers by meaning
- Recurrent Video Restoration Transformer with Guided Deformable Attention (NeurIPS 2022 · similarity 0.88)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.85)
- History-Guided Video Diffusion (ICML 2025 · similarity 0.84)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.84)
- RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO (NeurIPS 2026 · similarity 0.84)
- FMA-Net: Flow-Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.