Research map / Video / Video segmentation and tracking
SVFR: A Unified Framework for Generalized Video Face Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SVFR, a unified video diffusion framework built on Stable Video Diffusion that jointly handles video blind face restoration, colorization and inpainting, using task embeddings, unified latent regularization, facial prior learning and self-referred refinement.
Problem
Video face restoration lags image face restoration, suffering from temporal jitter, low fidelity and scarce high-quality video data; existing methods handle only blind face restoration, ignoring related subtasks like colorization and inpainting.
Approach
Treat blind face restoration, colorization and inpainting as one generalized video face restoration task solved by a single SVD-based diffusion model conditioned on the degraded video, a learnable task embedding, and a reference identity frame; a contrastive unified latent regularization aligns features across tasks, a landmark-based facial prior loss injects face structure, and self-referred refi
Key results
- On VFHQ-test BFR, SVFR reaches PSNR 29.563, LPIPS 0.223 and FVD 89.316, beating PGTFormer (28.996 PSNR, FVD 154.875).
- A single unified SVFR model gives colorization PSNR 23.079 and inpainting PSNR 29.119, exceeding the per-task GPEN, CodeFormer and PGDiff baselines.
- Ablation shows multi-task training plus ULR and FPL improves BFR PSNR from 28.323 (single-task) to 29.563.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train jointly on BFR, colorization, and inpainting rather than training each task separately (ablation) | SVFR · VFHQ-test BFR | PSNR: 28.323 -> 28.936 | Helped |
| train jointly on BFR, colorization, and inpainting rather than training each task separately (ablation) | SVFR · VFHQ-test colorization | PSNR: 22.233 -> 22.921 | Helped |
| train jointly on BFR, colorization, and inpainting rather than training each task separately (ablation) | SVFR · VFHQ-test inpainting | PSNR: 27.765 -> 28.303 | Helped |
| add Unified Latent Regularization and facial prior learning (ablation) | SVFR multi-task model · VFHQ-test BFR | PSNR: 28.936 -> 29.563 | Helped |
| add Unified Latent Regularization and facial prior learning (ablation) | SVFR multi-task model · VFHQ-test colorization | PSNR: 22.921 -> 23.079 | Helped |
| add Unified Latent Regularization and facial prior learning (ablation) | SVFR multi-task model · VFHQ-test inpainting | PSNR: 28.303 -> 29.119 | Helped |
| pretrain with task-specific prior knowledge rather than generative StyleGAN2 pretraining (ablation) | GPEN · VFHQ video BFR | FID: 30.023 -> 28.014 | Helped |
| pretrain with task-specific prior knowledge rather than generative StyleGAN2 pretraining (ablation) | GPEN · VFHQ video colorization | FID: 12.677 -> 11.941 | Helped |
| pretrain with task-specific prior knowledge rather than generative StyleGAN2 pretraining (ablation) | GPEN · VFHQ video inpainting | FID: 9.907 -> 9.089 | Helped |
| use self-referred refinement between sequential video segments (ablation) | SVFR · long video restoration | temporal consistency | Helped |
Surprising findings
- Pilot study shows transfer learning from related subtasks (e.g., inpainting pretraining for BFR) beats generative StyleGAN2 pretraining across all three tasks.
Limitations
- Relies on training data filtered from VoxCeleb2, CelebV-Text and VFHQ; high-quality video face data remains scarce.
- Self-referred refinement depends on a reference frame chosen from earlier generated segments, so long-sequence consistency relies on this design choice.
Experimental setup
Datasets: VoxCeleb2 CelebV-Text VFHQ
Benchmarks: VFHQ-test
Metrics: PSNR SSIM LPIPS IDS VIDD FVD
Baselines: GPEN CodeFormer PGDiff PGTFormer KEEP
Builds on: Stable Video Diffusion GPEN ControlNet-style conditioning
Closest papers by meaning
- URWKV: Unified RWKV Model with Multi-state Perspective for Low-light Image Restoration (CVPR 2025 · similarity 0.83)
- Dynamic Content Prediction with Motion-aware Priors for Blind Face Video Restoration (CVPR 2025 · similarity 0.82)
- Discrete Prior-Based Temporal-Coherent Content Prediction for Blind Face Video Restoration (AAAI 2025 · similarity 0.81)
- UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Prior (CVPR 2025 · similarity 0.81)
- FMA-Net: Flow-Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring (CVPR 2024 · similarity 0.81)
- Navigating Image Restoration with VAR's Distribution Alignment Prior (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.