Research map / Video / Video segmentation and tracking
Instant Video Models: Universal Adapters for Stabilizing Image-Based Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes causal stability adapters (EMA-style feature and output stabilizers with learned, spatially fused decay controllers) trained on frozen frame-based networks with a unified accuracy-stability-robustness loss to improve temporal stability and corruption robustness.
Problem
Frame-wise image models produce temporally inconsistent (flickering) outputs on video, and this worsens under transient time-varying input corruptions.
Approach
Lightweight causal stabilizers inserted at intermediate layers and outputs of a frozen image network, with decay values predicted by a stabilization controller (optionally with spatial fusion kernels), trained with a unified accuracy-stability-robustness loss whose oracle and collapse bounds on the stability weight lambda are derived theoretically.
Key results
- Spatial-fusion stabilizer for HDRNet reduces frame-to-frame instability by up to about 35% while exceeding base-model PSNR, with roughly 2 dB gain for the high-intensity effect.
- Setting lambda above the collapse bound (lambda=8 > tau-1) yields prediction collapse with instability below 1e-3 for enhancement and denoising.
- Controlled stabilizers for NAFNet improve both stability and PSNR when lambda <= 0.4 on denoising; on adverse weather, stabilizers substantially improve PSNR and stability over the unstabilized baseline, with joint fine-tuning plus stabilization giving the best results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add controlled stabilization with spatial fusion | HDRNet · NFS image enhancement | PSNR: roughly 2~dB for the high-intensity effect | Helped |
| Add spatial-fusion stabilization | HDRNet image enhancement · NFS image enhancement | instability: up to $\approx 35\%$ | Helped |
| Use controlled or spatial stabilization instead of the unstabilized model | HDRNet · NFS image enhancement | PSNR | Helped |
| Add feature-space stabilization with a static stabilizer (ablation) | HDRNet · NFS image enhancement | PSNR | No clear effect |
| Increase stabilization with a static stabilizer (ablation) | HDRNet · NFS image enhancement | PSNR | Hurt |
| Add a learned controller that targets temporally smooth features (ablation) | NAFNet · NFS denoising | PSNR | Helped |
| Apply fixed EMA stabilization to the features and output (ablation) | NAFNet · NFS denoising | PSNR | Hurt |
| Add spatial fusion to controlled stabilization (ablation) | NAFNet · NFS denoising, extreme noise | PSNR: about 6~dB worse | Hurt |
| Train spatial-fusion stabilizers with a frozen base model | NAFNet · RobustSpring rain and snow | PSNR | Helped |
| Fine-tune the base model on weather corruptions without stabilizers (ablation) | NAFNet · RobustSpring rain and snow | PSNR | Helped |
| Fine-tune the base model on weather corruptions without stabilizers (ablation) | NAFNet · RobustSpring rain and snow | instability | Hurt |
| Jointly train the stabilizer and base-model parameters (ablation) | NAFNet · RobustSpring rain and snow | PSNR | Helped |
Surprising findings
- Naive feature-space stabilization of the simple fixed type worsens both PSNR and instability for denoising, because the network's noise residual is uncorrelated across frames.
- Spatial-fusion stabilizer for NAFNet is about 6 dB worse on extreme-noise long sequences (hundreds of frames), though this gap shrinks with shorter tau.
Limitations
- Theoretical oracle and collapse bounds only hold when the distance delta is a norm; many multi-component losses are excluded.
- Sim-to-real gap when training depth stabilizers on simulated VisionSim video, with unmodeled baseline corruptions in real video.
- Optimal transport metric alternative is slow (seconds per loss evaluation), making it a training bottleneck.
Experimental setup
Datasets: Need for Speed (NFS) DAVIS VisionSim-generated indoor depth videos RobustSpring VIPER
Benchmarks: Need for Speed (image enhancement and denoising) RobustSpring (rain and snow) VisionSim depth validation VIPER (semantic segmentation)
Metrics: PSNR SSIM instability AbsRel Delta-1.25 pixel accuracy mIoU categorical instability
Baselines: simple fixed EMA output fixed EMA simple learned EMA blind video temporal consistency methods
Builds on: exponential moving average Clockwork ConvNets Hendrycks and Dietterich corruption robustness
Closest papers by meaning
- Single-Frame Regularization for Temporally Stable CNNs (CVPR 2019 · similarity 0.89)
- Improving the Robustness of Deep Neural Networks via Stability Training (CVPR 2016 · similarity 0.89)
- Robust Video Stabilization by Optimization in CNN Weight Space (CVPR 2019 · similarity 0.87)
- Harnessing Meta-Learning for Improving Full-Frame Video Stabilization (CVPR 2024 · similarity 0.86)
- ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object (CVPR 2024 · similarity 0.85)
- Robust-R1: Degradation-Aware Reasoning for Robust Visual Understanding (AAAI 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.