Research map / Video / Video segmentation and tracking
Efficient Video Face Enhancement with Enhanced Spatial-Temporal Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an efficient two-stage 3D-VQGAN with spatial-temporal codebooks and marginal prior regularization for blind video face restoration and video de-flickering.
Problem
Blind video face enhancement suffers from long processing times and inconsistent spatial-temporal effects such as flickering in restored face videos.
Approach
A 3D-VQGAN backbone with separate spatial and motion-aware temporal codebooks, a DINOv2-based multi-head discriminator, and a marginal prior KL regularizer that counts code usage via similarity scores to avoid codebook collapse; Stage II trains transformers to predict code indices from low-quality videos.
Key results
- On VFHQ-Test BFVR, the method reaches 27.47 PSNR, 0.8641 SSIM and 2.995 s runtime per 24-frame clip, versus 7.085 s for PGTFormer, while PGTFormer has higher PSNR (28.68).
- Brightness de-flickering FVD 100.7 and Flow-Score 1.100 vs. NeuralAtlas 111.0 and 1.468, with runtime 2.934 s vs. 851.3 s.
- Marginal prior regularization raises codebook-reconstruction SSIM to 0.9054 and lowers FVD to 49.11, versus 0.9015/56.46 for PDR and 0.4667/940.1 for CVQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed method instead of PGTFormer | Blind face video restoration · VFHQ-Test | SSIM: 0.8426 -> 0.8641 | Helped |
| Use the proposed method instead of PGTFormer | Blind face video restoration · VFHQ-Test | FVD: 107.6 -> 105.1 | Helped |
| Use the proposed method instead of PGTFormer | Blind face video restoration · VFHQ-Test | PSNR: 28.68 -> 27.47 | Hurt |
| Use the proposed method instead of DVP | Brightness de-flickering · Brightness de-flickering | Flow-Score: 1.492 -> 1.100 | Helped |
| Use the proposed method instead of DVP | Pixel de-flickering · Pixel de-flickering | Flow-Score: 1.505 -> 1.063 | Helped |
| Use ViT-B CLIP rather than ViT-S DINOv2 as discriminator feature network (ablation) | Stage I reconstruction · VFHQ-Test reconstruction | SSIM: 0.9054 -> 0.8935 | Hurt |
| Fuse spatial and temporal latents with convolution projection instead of addition (ablation) | Stage I reconstruction · VFHQ-Test reconstruction | SSIM: 0.9054 -> 0.8799 | Hurt |
| Fuse spatial and temporal latents with 3DFFT instead of addition (ablation) | Stage I reconstruction · VFHQ-Test reconstruction | SSIM: 0.9054 -> 0.8741 | Hurt |
| Add adversarial loss (ablation) | Stage I reconstruction · VFHQ-Test reconstruction | SSIM: 0.8887 -> 0.8914 | Helped |
| Add marginal prior regularization (ablation) | Stage I reconstruction · VFHQ-Test reconstruction | SSIM: 0.8914 -> 0.9054 | Helped |
| Use MPR rather than no codebook regularization (ablation) | Stage I codebooks · VFHQ-Test reconstruction | SSIM: 0.8914 -> 0.9054 | Helped |
| Use MPR rather than PDR (ablation) | Stage I codebooks · VFHQ-Test reconstruction | SSIM: 0.9015 -> 0.9054 | Helped |
Surprising findings
- PGTFormer achieves higher PSNR (28.68) than the proposed method on VFHQ-Test, so the method is not best on all fidelity metrics.
- Continuous vector quantization (CVQ) collapsed when jointly optimizing multiple codebooks (SSIM 0.4667).
Limitations
- Improvement in PSNR over PGTFormer is absent; PGTFormer leads on PSNR and LPIPS-related fidelity in some comparisons.
- Training data includes a self-collected live broadcast dataset that is not released.
Experimental setup
Datasets: VFHQ live broadcast dataset (self-collected)
Benchmarks: VFHQ-Test
Metrics: PSNR SSIM LPIPS AKD Face-Cons IDS FVD Flow-Score Runtime
Baselines: VQFR GFPGAN CodeFormer BasicVSR++ Real-BasicVSR PGTFormer KEEP FastBlend DVP NeuralAtlas
Builds on: VQGAN VQFR CodeFormer DINOv2
Closest papers by meaning
- Discrete Prior-Based Temporal-Coherent Content Prediction for Blind Face Video Restoration (AAAI 2025 · similarity 0.87)
- Dynamic Content Prediction with Motion-aware Priors for Blind Face Video Restoration (CVPR 2025 · similarity 0.86)
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.84)
- PMQ-VE: Progressive Multi-Frame Quantization for Video Enhancement (NeurIPS 2025 · similarity 0.84)
- Blind Face Restoration under Extreme Conditions: Leveraging 3D-2D Prior Fusion for Superior Structural and Texture Recovery (AAAI 2024 · similarity 0.84)
- Occlusion-Insensitive Talking Head Video Generation via Facelet Compensation (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.