Research map / Video / Video segmentation and tracking
Blind Bitstream-corrupted Video Recovery via Metadata-guided Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M-GDM, a metadata-guided latent video diffusion model that recovers bitstream-corrupted video without predefined masks, using motion vectors and frame types plus a learned pseudo-mask predictor and post-refinement module.
Problem
Bitstream-corrupted video recovery currently requires manually annotated corruption masks at inference, which is laborious and impractical; the paper targets blind recovery where corrupted regions must be located automatically.
Approach
Uses video metadata (H.264 motion vectors and frame types) as a corruption indicator via a dual-stream encoder injected through cross-attention into a video latent diffusion U-Net; a prior-driven mask predictor built from diffusion attention and metadata features produces pseudo masks for hard compositing, and a Swin-based post-refinement module with a temporal PatchGAN removes boundary artifacts.
Key results
- On YouTube-VOS, M-GDM reaches 28.21 PSNR and 0.0333 LPIPS, beating BSCVR (27.68 PSNR) without any input masks.
- On DAVIS, M-GDM reaches 26.05 PSNR and 0.1621 VFID, exceeding BSCVR by 1.55 dB PSNR.
- Ablation on DAVIS: adding the metadata encoder raises PSNR from 21.56 to 22.27, and the mask predictor raises it to 25.84.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M-GDM instead of BSCVR | Blind bitstream-corrupted video recovery · YouTube-VOS | PSNR: 27.68 -> 28.21 | Helped |
| Use M-GDM instead of BSCVR | Blind bitstream-corrupted video recovery · DAVIS | PSNR: 24.50 -> 26.05 | Helped |
| Add the dual-stream metadata encoder (DME) (ablation) | Fine-tuned temporal U-Net · DAVIS | PSNR: 21.56 -> 22.27 | Helped |
| Add the dual-stream metadata encoder (DME) (ablation) | Fine-tuned temporal U-Net · DAVIS | LPIPS: 0.0565 -> 0.0538 | Helped |
| Add the prior-driven mask predictor (PMP) (ablation) | Temporal U-Net + DME · DAVIS | PSNR: 22.27 -> 25.84 | Helped |
| Add the prior-driven mask predictor (PMP) (ablation) | Temporal U-Net + DME · DAVIS | LPIPS: 0.0538 -> 0.0328 | Helped |
| Add the post-refinement module (PRM) (ablation) | Temporal U-Net + DME + PMP · DAVIS | PSNR: 25.84 -> 26.05 | Helped |
| Add the post-refinement module (PRM) (ablation) | Temporal U-Net + DME + PMP · DAVIS | LPIPS: 0.0328 -> 0.0340 | Hurt |
Surprising findings
- Adding the post-refinement module raised PSNR (25.84 to 26.05) but slightly worsened LPIPS (0.0328 to 0.0340).
- The authors note a slight color deviation from the original corrupted videos, attributed to the video diffusion backbone designed for generation.
Limitations
- Slight color deviation from the corrupted input, since the backbone is a video generation model rather than one designed for recovery.
- Baselines were given SAM2-generated pseudo masks rather than ground-truth masks, so the comparison depends on the quality of that mask generator.
- Evaluation is limited to two BSCV validation subsets of 508 and 50 clips, with no reported runtime or efficiency numbers.
Experimental setup
Datasets: BSCV YouTube-VOS DAVIS
Benchmarks: BSCV YouTube-VOS validation BSCV DAVIS validation
Metrics: PSNR SSIM LPIPS VFID
Baselines: E2FGVI ProPainter BSCVR
Builds on: latent diffusion model Stable Video Diffusion-style temporal U-Net BSCVR SwinIR residual swin transformer blocks ProPainter
Closest papers by meaning
- BlurDM: A Blur Diffusion Model for Image Deblurring (NeurIPS 2025 · similarity 0.84)
- Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model (NeurIPS 2024 · similarity 0.83)
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.83)
- AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration (ICML 2025 · similarity 0.83)
- One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution (NeurIPS 2025 · similarity 0.83)
- OSCAR: One-Step Diffusion Codec Across Multiple Bit-rates (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.