Research map / Video / Video generation
Classic Video Denoising in a Machine Learning World: Robust, Fast, and Controllable
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RFCVD, a video denoiser that wraps a differentiable classic pipeline (Wiener temporal filter plus bilateral Laplacian pyramid spatial filter) whose parameters are predicted by a ConvNext-based hypernetwork from an anchor frame, trained with AWGN+H.264 augmentation.
Problem
Learned video denoisers are brittle on real-world noise patterns outside their training distribution, are slow, and offer no user control, while classic denoisers are robust and fast but require manual per-video parameter tuning.
Approach
Implement a traditional video denoising pipeline differentiably and train a hypernetwork to predict its spatially-varying parameters from a single anchor frame's noise profile, so noise is analyzed once per video; scalar knobs on the predicted parameters give user control.
Key results
- On CRVD (sRGB) across ISO 1600-25600, RFCVD achieves 36.04 average PSNR, ranked 1st of 10, versus 35.32 for retrained NAFNet.
- RFCVD runs at 31.66 FPS on an RTX 3090, about four times faster than the second-fastest method (SID† at 6.95 FPS).
- When trained only on AWGN+H.264, RFCVD ranks first on both synthetic test degradations in PSNR and SSIM (e.g., 28.59 PSNR on film grain/AV1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace spatially varying denoising parameters with scalar parameters (ablation) | RFCVD · CRVD (sRGB) | PSNR: 36.04 -> 33.62 | Hurt |
| Remove image gradients from the denoising-parameter predictor input (ablation) | RFCVD · CRVD (sRGB) | PSNR: 36.04 -> 35.61 | Hurt |
| Remove the alignment-validity mask from the denoising-parameter predictor input (ablation) | RFCVD · CRVD (sRGB) | PSNR: 36.04 -> 35.26 | Hurt |
| Remove backbone B from the hypernetwork (ablation) | RFCVD · CRVD (sRGB) | PSNR: 36.04 -> 34.38 | Hurt |
| Do not fine-tune backbone B (ablation) | RFCVD · CRVD (sRGB) | PSNR: 36.04 -> 35.71 | Hurt |
| Remove NPA stabilization (ablation) | RFCVD · CRVD (sRGB) | PSNR: 36.04 -> 35.83 | Hurt |
| Remove the consistency loss (ablation) | RFCVD with first-frame anchor · CRVD (sRGB) | PSNR: 36.04 -> 35.80 | Hurt |
| Fine-tune estimated denoising parameters on ground truth (ablation) | RFCVD · CRVD (sRGB) | PSNR: 36.04 -> 36.75 | Helped |
| Use RFCVD instead of TOFlow† | Video denoising · Film grain noise with AV1 compression | PSNR: 28.15 -> 28.59 | Helped |
| Use RFCVD instead of VRT† | Video denoising · Spatially correlated noise with H.265 compression | PSNR: 28.78 -> 28.93 | Helped |
| Use RFCVD instead of BasicVSR++† | Video denoising · CRVD (sRGB), overall | FPS: 7.41 -> 31.66 | Helped |
Surprising findings
- Retraining existing baselines with the simple AWGN+H.264 augmentation substantially improved them on CRVD (e.g., NAFNet +5.07 dB, FastDVDNet +6.75 dB).
- Choice of anchor frame (first, middle, last) had negligible effect on denoising quality.
Limitations
- Gap to an oracle: fine-tuning the predicted parameters on ground truth improves PSNR by 0.70 dB, so estimates are not optimal.
- Hypernetworks are not well supported by common inference runtimes (CoreML, WinML, OpenVINO), complicating deployment.
- When the noise profile is known in advance, learned methods would likely perform better and a baked-in model could be faster.
Experimental setup
Datasets: REDS CRVD
Benchmarks: CRVD (sRGB) REDS validation with film grain/AV1 and spatially correlated noise/H.265
Metrics: PSNR SSIM LPIPS FPS
Baselines: NAFNet SID FastDVDNet TOFlow BasicVSR++ VRT Real-ESRGAN UDVD MF2F
Builds on: NPA SpyNet ConvNext Hasinoff et al. multi-frame burst denoising (Wiener merge) multiresolution bilateral filtering
Closest papers by meaning
- FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation (CVPR 2020 · similarity 0.90)
- Real-Time Controllable Denoising for Image and Video (CVPR 2023 · similarity 0.90)
- IDR: Self-Supervised Image Denoising via Iterative Data Refinement (CVPR 2022 · similarity 0.89)
- Model-Blind Video Denoising via Frame-To-Frame Training (CVPR 2019 · similarity 0.88)
- Unsupervised Deep Video Denoising with Untrained Network (AAAI 2023 · similarity 0.88)
- Supervised Raw Video Denoising With a Benchmark Dataset on Dynamic Scenes (CVPR 2020 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.