Research map / Video / Video segmentation and tracking
VideoSPatS: Video SPatiotemporal Splines for Disentangled Occlusion, Appearance and Motion Modeling and Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoSPatS, an implicit video representation using spatial and color spline deformation fields (neural spline fields) to disentangle occlusion, appearance and motion from monocular video, enabling consistent canonical-space editing.
Problem
Implicit video representations struggle to disentangle motion, time-varying appearance (highlights, shadows) and occlusions, producing distorted or static canonical images that hinder consistent video editing.
Approach
Map coordinates to spatial and color spline control points (cubic Hermite splines) so that motion and time-varying color are separately modeled, with per-layer (foreground occluder / background) branches and an alpha matte; editing is applied to the base-color canonical space and propagated via the splines.
Key results
- On DAVIS editing warping consistency, outperforms CoDeF by 4.66 dB and Deformable Sprites by 0.4 dB.
- Trains a 50-frame 512x288 video in about 90 minutes on a single A10G GPU (~4 GB).
- Ablations show removing color splines or optical flow loss degrades canonical spaces and disentanglement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoSPatS instead of CoDeF | Video modeling on DAVIS · DAVIS | warping consistency: +4.66dB | Helped |
| Use VideoSPatS instead of Def. Sprites | Video modeling on DAVIS · DAVIS | warping consistency: +0.4dB | Helped |
| Add spatial and color spline deformation fields (ablation) | VideoSPatS · Video modeling | canonical-space quality | Helped |
| Add color spline deformation while retaining spatial splines (ablation) | VideoSPatS with spatial splines · Video modeling | motion and appearance disentanglement | Helped |
| Add spatial spline deformation while retaining color splines (ablation) | VideoSPatS with color splines · Video modeling | render quality | Helped |
| Add color deformation splines (ablation) | VideoSPatS with spatial splines · Video modeling with changing appearance | render quality | Helped |
| Add optical flow loss (ablation) | VideoSPatS · Video modeling with large deformations | motion consistency | Helped |
| Add the MXE loss to the layer separation objective (ablation) | VideoSPatS · Occlusion separation | alpha-mask refinement | Helped |
Surprising findings
- CoDeF achieves slightly better reconstruction than VideoSPatS on DAVIS even though VideoSPatS yields more editable canonical spaces.
Limitations
- Restricted by the extent of deformation and self-occlusions present in input videos.
- Requires user-provided coarse masks for foreground/background layer separation.
Experimental setup
Datasets: DAVIS CoDeF dataset YT-in-the-wild (12 Creative Commons YouTube clips)
Benchmarks: DAVIS CoDeF dataset YT-in-the-wild
Metrics: warping consistency (dB) for video editing qualitative canonical space comparison
Baselines: CoDeF Deformable Sprites
Builds on: Neural Spline Fields CoDeF Neural Atlases RAFT
Closest papers by meaning
- MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion (ICLR 2025 · similarity 0.86)
- Generative Omnimatte: Learning to Decompose Video into Layers (CVPR 2025 · similarity 0.86)
- DisMo: Disentangled Motion Representations for Open-World Motion Transfer (NeurIPS 2025 · similarity 0.86)
- Taming generative video models for zero-shot optical flow extraction (NeurIPS 2025 · similarity 0.85)
- Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices (ICML 2024 · similarity 0.85)
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.