Research map / Video / Video segmentation and tracking
Discrete Prior-Based Temporal-Coherent Content Prediction for Blind Face Video Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP-TempCoh, a transformer that predicts discrete visual-prior tokens from degraded face video tokens and modulates them with discrete motion statistics priors to improve temporal coherence in blind face video restoration.
Problem
Blind face video restoration must handle unknown degradations and temporal heterogeneity while keeping face identity and attributes stable across frames; image-based restorers and diffusion samplers produce flicker.
Approach
Combines a spatial-temporal-aware transformer that predicts indices into a learned visual codebook (latent bank) with a motion statistics bank that stores cross-frame channel mean/variance, used to re-normalize predicted features so their statistics match real face videos.
Key results
- On VFHQ-Test-Deg, DP-TempCoh reaches PSNR 25.12, FID 52.04 and IFD 3.80, outperforming CodeFormer (FID 65.68, IFD 9.86) and the best video baseline.
- On HDTF-Deg, DP-TempCoh achieves FID 25.65 and IFD 3.13, versus CodeFormer FID 34.14 among the second-best methods.
- Ablation: adding motion statistics modulation to the S&T-aware module lowers IFD from 3.92 to 3.80 while also improving PSNR and FID.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace spatial-temporal-aware prediction with spatial-aware prediction (ablation) | DP-TempCoh · VFHQ-Test-Deg | PSNR: 24.52 -> 23.12 | Hurt |
| Replace spatial-aware prediction with spatial-temporal-aware prediction (ablation) | DP-TempCoh · VFHQ-Test-Deg | IFD: 9.86 -> 3.92 | Helped |
| Add motion statistics modulation to spatial-temporal-aware content prediction (ablation) | DP-TempCoh · VFHQ-Test-Deg | PSNR: 24.52 -> 25.12 | Helped |
| Add motion statistics modulation to spatial-temporal-aware content prediction (ablation) | DP-TempCoh · VFHQ-Test-Deg | FID: 55.11 -> 52.04 | Helped |
| Add motion statistics modulation to spatial-temporal-aware content prediction (ablation) | DP-TempCoh · VFHQ-Test-Deg | IFD: 3.92 -> 3.80 | Helped |
| Use motion modulation without content prediction instead of spatial-aware prediction (ablation) | DP-TempCoh · VFHQ-Test-Deg | IFD: 9.86 -> 5.59 | Helped |
Surprising findings
- The motion-only variant (c) yields very low PSNR (12.55) and high FID (210.12) despite reducing IFD, indicating the modulation alone is insufficient without content prediction.
Limitations
- Relies on synthetic degradation for training; generalization to natural degradations is assessed only on YTF subsets.
- Some competing methods (e.g., CodeFormer) achieve lower LPIPS or FID on certain metrics, and the paper notes trade-offs in PSNR and FID for individual ablations.
Experimental setup
Datasets: VFHQ HDTF VFHQ-Test YouTube Faces
Benchmarks: VFHQ-Test-Deg HDTF-Deg YTF-Medium YTF-Hard
Metrics: PSNR IDS LPIPS FID IFD user study scores
Baselines: CodeFormer RestoreFormer DR2 DiffBIR DifFace BasicVSR++ MIA-VSR IA-RT FMA-Net
Builds on: VQGAN CodeFormer BasicVSR++
Closest papers by meaning
- Dynamic Content Prediction with Motion-aware Priors for Blind Face Video Restoration (CVPR 2025 · similarity 0.93)
- Efficient Video Face Enhancement with Enhanced Spatial-Temporal Consistency (CVPR 2025 · similarity 0.87)
- Blind Face Restoration under Extreme Conditions: Leveraging 3D-2D Prior Fusion for Superior Structural and Texture Recovery (AAAI 2024 · similarity 0.86)
- $InterLCM$: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restoration (ICLR 2025 · similarity 0.84)
- ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration (NeurIPS 2024 · similarity 0.84)
- A Modular Conditional Diffusion Framework for Image Reconstruction (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.