Research map / Video / Video segmentation and tracking
WaveFormer: Wavelet Transformer for Noise-Robust Video Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WaveFormer, a wavelet-domain spatial-temporal transformer that computes attention only on DWT low-frequency components (treating high-frequency components as noise) to improve video inpainting on YouTube-VOS and DAVIS.
Problem
Video inpainting with transformers relies on patch-wise attention retrieval, which noise in embeddings corrupts, leading to blurry or misplaced completions.
Approach
Apply a Discrete Wavelet Transform to queries, keys and values so attention is computed only from clean low-frequency sub-bands, then share the attention weights with high-frequency sub-bands and reconstruct via inverse DWT; a theorem is offered claiming noise distorts softmax attention weights.
Key results
- On YouTube-VOS, WaveFormer reaches 33.264 PSNR and 0.1184 Ewarp versus 30.811 PSNR for the best prior baseline FGT.
- On DAVIS, WaveFormer reaches 34.169 PSNR and 0.9475 SSIM, versus 32.742 PSNR for FGT.
- Applying DWT-based attention to STTN and FuseFormer improves PSNR by about 1.0 and 1.4 dB respectively (STTN Wave 30.012, FuseFormer Wave 31.171).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use WaveFormer instead of FGT | video inpainting · YouTube-VOS | PSNR: 30.811 -> 33.264 | Helped |
| use WaveFormer instead of FGT | video inpainting · YouTube-VOS | Ewarp: 0.1308 -> 0.1184 | Helped |
| use WaveFormer instead of OPN | video inpainting · YouTube-VOS | LPIPS: 0.4145 -> 0.2933 | Helped |
| use WaveFormer instead of FGT | video inpainting · DAVIS | PSNR: 32.742 -> 34.169 | Helped |
| use WaveFormer instead of OPN | video inpainting · DAVIS | SSIM: 0.9302 -> 0.9475 | Helped |
| use WaveFormer instead of OPN | video inpainting · DAVIS | LPIPS: 0.3876 -> 0.3137 | Helped |
| add DWT-based frequency separation to attention computation (ablation) | WaveFormer · Video inpainting evaluation (dataset not stated) | PSNR: 31.326 -> 33.264 | Helped |
| add DWT-based frequency separation to attention computation (ablation) | WaveFormer · Video inpainting evaluation (dataset not stated) | SSIM: 0.9259 -> 0.9435 | Helped |
| add DWT-based frequency separation to attention computation (ablation) | WaveFormer · Video inpainting evaluation (dataset not stated) | Ewarp: 0.1299 -> 0.1184 | Helped |
| add DWT-based frequency separation to attention computation (ablation) | WaveFormer · Video inpainting evaluation (dataset not stated) | LPIPS: 0.3471 -> 0.2933 | Helped |
| apply DWT noise separation | STTN · Video inpainting evaluation (dataset not stated) | PSNR: 28.993 -> 30.012 | Helped |
| apply DWT noise separation | FuseFormer · Video inpainting evaluation (dataset not stated) | PSNR: 29.765 -> 31.171 | Helped |
Surprising findings
- The paper reports that E2FGVI's Ewarp on DAVIS (0.4579) is far worse than other methods, an outlier relative to its PSNR.
Limitations
- The noise-robustness theorem is argued via a simplified attention model with bounded features, and the paper does not show how it transfers to trained networks beyond feature visualizations.
- Evaluation on only two benchmarks with simulated noise and masks; no released code or weights are stated.
Experimental setup
Datasets: YouTube-VOS DAVIS FVI
Benchmarks: YouTube-VOS DAVIS
Metrics: PSNR SSIM Ewarp LPIPS
Baselines: TCCDS VINet DFVI FGVC CPVINet OPN STTN FuseFormer E2FGVI FGT
Builds on: STTN FuseFormer Temporal PatchGAN
Closest papers by meaning
- Video Diffusion Models Are Strong Video Inpainter (AAAI 2025 · similarity 0.87)
- Keyframe-Guided Creative Video Inpainting (CVPR 2025 · similarity 0.84)
- SyFormer: Structure-Guided Synergism Transformer for Large-Portion Image Inpainting (AAAI 2024 · similarity 0.84)
- AVID: Any-Length Video Inpainting with Diffusion Model (CVPR 2024 · similarity 0.84)
- Parameterized Stripe Attention for Efficient Video Generation (NeurIPS 2026 · similarity 0.83)
- HomoGen: Enhanced Video Inpainting via Homography Propagation and Diffusion (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.