Research map / Video / Video segmentation and tracking
Splatter a Video: Video Gaussian Representation for Versatile Processing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Represents a monocular video as dynamic 3D Gaussians in an orthographic camera space, supervised by distilled optical flow and depth priors plus a rigidity loss, and applies the representation to tracking, depth/feature consistency, editing, interpolation, novel view and stereo video generation.
Problem
Video representations for processing either struggle with complex motion and occlusion (2D/2.5D atlases, canonical images) or are implicit 3D fields ill-suited for explicit manipulation; monocular videos lack the 3D information needed to fit a consistent 3D structure.
Approach
Embed a video into explicit 3D Gaussians in a canonical orthographic camera space, give each Gaussian polynomial+Fourier motion coefficients, and regularize learning with distilled RAFT optical flow, Marigold depth, segmentation masks, and an as-rigid-as-possible loss, so that the representation supports explicit editing and 3D-aware processing.
Key results
- Average PSNR of 28.44 on a DAVIS subset (7 videos), versus 27.75 for CoDeF and 24.07 for OmniMotion.
- Removing the rigid loss reduces reconstruction PSNR by 1.51 dB according to the ablation.
- Trained on a single NVIDIA 3090 in about 15-20 minutes per video.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use video Gaussian representation instead of CoDeF | Video reconstruction · Average across Bear, Cow, Elephant, Breakdance-Flare, Train, Camel, and Kite-surf | PSNR: 27.75 -> 28.44 | Helped |
| Use video Gaussian representation instead of CoDeF | Video reconstruction · Bear, Cow, Elephant, Breakdance-Flare, Train, Camel, and Kite-surf | PSNR: 29.17 -> 30.17; 28.82 -> 28.24; 30.50 -> 29.82; 25.99 -> 27.18; 26.53 -> 28.09; 26.10 -> 27.74; 27.17 -> 27.82 | Mixed |
| Add rigid motion constraint (ablation) | Video Gaussian representation · Video reconstruction | PSNR: 1.51 dB | Helped |
| Add depth prior (ablation) | Video Gaussian representation · Novel view synthesis | novel view synthesis capability | Helped |
Surprising findings
- Without the depth prior, Gaussians collapse into a flat 2D plane, making the method resemble 2D layer-based methods.
Limitations
- Struggles with large scene changes and large deformations, which are hard to optimize.
- Relies on existing correspondence estimators such as RAFT, which can fail for rapid, highly non-rigid motion.
- Per-video optimization rather than a generalizable model.
Experimental setup
Datasets: DAVIS
Benchmarks: DAVIS
Metrics: PSNR
Baselines: OmniMotion CoDeF
Builds on: 3D Gaussian Splatting OmniMotion CoDeF Gaussian-Flow
Closest papers by meaning
- HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.88)
- Large Images Are Gaussians: High-Quality Large Image Representation with Levels of 2D Gaussian Splatting (AAAI 2025 · similarity 0.88)
- Fully Explicit Dynamic Gaussian Splatting (NeurIPS 2024 · similarity 0.88)
- SfM-Free 3D Gaussian Splatting via Hierarchical Training (CVPR 2025 · similarity 0.88)
- Gaussian Graph Network: Learning Efficient and Generalizable Gaussian Representations from Multi-view Images (NeurIPS 2024 · similarity 0.88)
- Motion Decoupled 3D Gaussian Splatting for Dynamic Object Representation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.