Research map / Video / Video segmentation and tracking
MR-VNet: Media Restoration using Volterra Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MR-VNet, a U-Net-style image/video restoration network built from second-order Volterra layers that replace activation functions with higher-order (quadratic) convolutions, and shows NAFNet is a rank-1 special case.
Problem
Image and video restoration (deblurring, denoising) needs strong non-linearity, but existing high-performing networks are heavy (transformers, multi-stage U-Nets); the paper asks whether Volterra-series higher-order convolutions can replace activation functions efficiently.
Approach
Replace activation functions with second-order Volterra kernels, implemented as lossless (symmetry-pruned) or lossy (rank-Q separable) approximations inside a U-Net; the rank-1 version reduces exactly to the NAFNet gate, so NAFNet is a special case.
Key results
- On GoPro deblurring, MR-VNet-LLA reaches 34.04 PSNR at 96 GMACs vs NAFNet 33.69 at 65 GMACs; the MR-VNet-LYA (Q=4) variant reaches 33.93 at 70 GMACs.
- On SIDD denoising, MR-VNet-LLA achieves 40.58 PSNR and 0.963 SSIM, above NAFNet's 40.30.
- On CRVD video denoising, MR-VNet-LYA reaches 41.93 PSNR, above LLVD-L at 41.41 with 117 GMACs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MR-VNet-LLA instead of NAFNet | Image restoration models · GoPro | PSNR: 33.69 -> 34.04 | Helped |
| Use MR-VNet-LLA instead of NAFNet | Image restoration models · SIDD | PSNR: 40.30 -> 40.58 | Helped |
| Use MR-VNet-LLA instead of NAFNet | Image restoration models · REDS | PSNR: 29.09 -> 29.92 | Helped |
| Use MR-VNet-LYA instead of LLVD-L | Video restoration models · CRVD | PSNR: 41.41 -> 41.93 | Helped |
| Increase Volterra kernel rank from Q=1 to higher ranks (ablation) | MR-VNet-LYA · GoPro and SIDD | PSNR: GoPro: 33.50 -> 33.85 -> 33.93 -> 33.97; SIDD: 40.21 -> 40.39 -> 40.43 -> 40.46 | Helped |
| Increase architecture width from 16 to 48 channels (ablation) | MR-VNet (Q=4) · GoPro | PSNR: 32.80 -> 32.92 -> 33.18 -> 33.93 | Helped |
| Add activation functions to the rank-1 NAFNet configuration (ablation) | NAFNet · SIDD | PSNR: 39.96 -> 39.98 (ReLU); 39.96 -> 39.97 (GELU); 39.96 -> 39.99 (Sigmoid); 39.96 -> 39.96 (SiLU) | Helped |
| Add activation functions to the higher-rank MR-VNet (ablation) | MR-VNet · SIDD | PSNR: 40.43 -> 40.40 (ReLU); 40.43 -> 40.39 (GELU); 40.43 -> 40.40 (Sigmoid); 40.43 -> 40.39 (SiLU) | Hurt |
Surprising findings
- Adding ReLU, GELU, Sigmoid or SiLU on top of the Volterra kernel yields no consistent gain (Table 8), suggesting higher-rank quadratic kernels already supply the non-linearity.
Limitations
- Quadratic-rank approximation gains over NAFNet are small on some benchmarks (e.g. about 0.1 dB on SIDD), and results for CRVD are reported only for the lossy variant.
- Video denoising uses only the lossy variant because the exact 3D kernel is computationally demanding.
- Proposition 3 (universal approximation via Volterra) is argued via a truncated Taylor expansion, so it is a loose justification.
Experimental setup
Datasets: GoPro REDS SIDD CRVD
Benchmarks: GoPro REDS SIDD CRVD
Metrics: PSNR SSIM GMACs
Baselines: MIMO-UNet HINet MAXIM Restormer UFormer DeepRFT MPRNet NAFNet MIR-Net NBNet EMVD-L RVIDEFormer
Builds on: Volterra filter NAFNet U-Net Volterra layers for action recognition
Closest papers by meaning
- A Restoration Network as an Implicit Prior (ICLR 2024 · similarity 0.86)
- Attention-Based Adaptive Selection of Operations for Image Restoration in the Presence of Unknown Combined Distortions (CVPR 2019 · similarity 0.86)
- Restore From Restored: Video Restoration With Pseudo Clean Video (CVPR 2021 · similarity 0.86)
- Functional Neural Networks for Parametric Image Restoration Problems (NeurIPS 2021 · similarity 0.86)
- Conquering the CNN Over-Parameterization Dilemma: A Volterra Filtering Approach for Action Recognition (AAAI 2020 · similarity 0.86)
- Omni-Kernel Network for Image Restoration (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.