Research map / Video / Video segmentation and tracking
Video Harmonization with Triplet Spatio-Temporal Variation Patterns
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Video Triplet Transformer (VTT) with short-term spatial, long-term global (masked prediction), and long-term dynamic (deformable attention) modules for video harmonization, plus a refined temporal consistency metric (fR-RTC), evaluated on HYouTube and transferred to video enhancement and demoiréing.
Problem
Video harmonization requires adjusting foreground appearance to match the background while keeping inter-frame consistency; image harmonization methods cause flickering, and existing temporal consistency metrics fail to capture abrupt flicker.
Approach
Model three spatio-temporal variation patterns (short-term spatial, long-term global with masked prediction, long-term dynamic with deformable attention) in a Transformer, and introduce a variance-amplified, foreground-focused relation-based temporal consistency metric (fR-RTC).
Key results
- On HYouTube, VHTT reaches 40.03 PSNR, 90.35 fMSE and 1.26 fR-RTC, outperforming CO2Net (37.61 PSNR, 186.72 fMSE, 6.07 fR-RTC).
- In a user study, VHTT was selected as best 61.5% of the time on HYouTube versus 13.5% for CO2Net and 25.0% for HT+.
- On video demoiréing, VHTT improves PSNR from 23.71 (VDRTC) to 28.56.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VHTT instead of CO2Net | Video harmonization · HYouTube test | fMSE: 186.72 -> 90.35 | Helped |
| Use VHTT instead of CO2Net | Video harmonization · HYouTube test | fR-RTC: 6.07 -> 1.26 | Helped |
| Apply VHTT rather than VEN-Retinex | Video enhancement · SDSD | PSNR: 26.10 -> 26.61 | Helped |
| Apply VHTT rather than VDRTC | Video demoiréing · Video demoiréing dataset | PSNR: 23.71 -> 28.56 | Helped |
| Add LT-GT to ST-ST (ablation) | VHTT · HYouTube test | fMSE: 111.59 -> 95.64 | Helped |
| Add LT-DT to ST-ST (ablation) | VHTT · HYouTube test | fMSE: 111.59 -> 105.24 | Helped |
| Add masked prediction to LT-GT (ablation) | VHTT · HYouTube test | fMSE: 99.97 -> 95.64 | Helped |
| Add masked prediction to LT-GT (ablation) | VHTT · HYouTube test | fMSE: 91.97 -> 90.35 | Helped |
| Use the complete triplet of spatial, global, and dynamic modules (ablation) | VHTT · HYouTube test | fMSE: 95.64 -> 90.35 | Helped |
| Use masked prediction rather than forward autoregressive prediction in LT-GT (ablation) | VHTT · HYouTube test | fMSE: 106.57 -> 95.64 | Helped |
| Use masked prediction rather than bidirectional autoregressive prediction in LT-GT (ablation) | VHTT · HYouTube test | fMSE: 104.05 -> 95.64 | Helped |
| Use masked prediction rather than separate forward/backward autoregressive prediction in LT-GT (ablation) | VHTT · HYouTube test | fMSE: 98.83 -> 95.64 | Helped |
Surprising findings
- Autoregressive (GPT-style) self-supervision for the long-term global module performed worse than vanilla self-attention, attributed to insufficient data.
- The standard relation-based temporal consistency metric (RTC) gives low values for all methods because most background pixels are unchanged, so it fails to separate methods; flow-based metrics gave 527 on real videos.
Limitations
- Higher FLOPs than several baselines (1727G vs 212G-385G for HT+/RainNet/BargainNet), and slower than HT+ and TCVHAN.
- Temporal consistency metrics and user study rely on limited test sets (HYouTube test set of 636 pairs).
Experimental setup
Datasets: HYouTube SDSD video demoiréing dataset (Chen et al. VDRTC)
Benchmarks: HYouTube SDSD video demoiréing dataset
Metrics: PSNR fPSNR fMSE R-RTC fR-RTC SSIM LPIPS RTC user study preference
Baselines: HT+ CO2Net TCVHAN IIH RainNet BargainNet VEN-Retinex VDRTC
Builds on: Transformer deformable attention masked autoencoder (MAE) relation-based temporal consistency metric
Closest papers by meaning
- FRIH: Fine-Grained Region-Aware Image Harmonization (AAAI 2024 · similarity 0.84)
- VideoSPatS: Video SPatiotemporal Splines for Disentangled Occlusion, Appearance and Motion Modeling and Editing (CVPR 2025 · similarity 0.84)
- MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion (ICLR 2025 · similarity 0.84)
- Training-and-Prompt-Free General Painterly Harmonization via Zero-Shot Disentenglement on Style and Content References (AAAI 2025 · similarity 0.84)
- Spectral Motion Alignment for Video Motion Transfer Using Diffusion Models (AAAI 2025 · similarity 0.83)
- Progressive Painterly Image Harmonization from Low-Level Styles to High-Level Styles (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.