Research map / 3D vision / 3D generation
DyST: Towards Dynamic Neural Scene Representations on Real-World Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DyST, a latent neural scene representation model that decomposes monocular real-world videos into scene content, per-view dynamics, and camera pose, trained by co-training on a new synthetic dataset DySO and SSv2 videos with a latent control swap scheme.
Problem
Learning latent 3D neural scene representations that capture both structure and dynamics from monocular real-world videos without ground-truth camera poses, with independent control over camera and scene dynamics.
Approach
A latent control swap training scheme on multi-view multi-dynamics synthetic scenes (DySO) forces camera latents and dynamics latents to be estimated from counterpart views, separating them; co-training with monocular SSv2 videos transfers this separation to real video, with no ground-truth poses needed.
Key results
- Latent control swap achieves PSNR 26.0 and LPIPS 0.34 on DySO, versus 18.6 PSNR for no swap.
- Camera contrastiveness R_cam = 0.06 and dynamics contrastiveness R_dyn = 0.42 with latent control swap, versus 0.72 and 1.26 without swap.
- Learned camera and dynamics latents on real SSv2 videos show distance patterns aligned with video events, and PCA interpolation yields meaningful camera and object motions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add latent control swapping instead of no swapping (ablation) | DyST · DySO | PSNR: 18.6 -> 26.0 | Helped |
| add latent control swapping instead of swapping with 50% probability (ablation) | DyST · DySO | PSNR: 25.4 -> 26.0 | Helped |
| add latent control swapping instead of averaging target and swapped-view latents (ablation) | DyST · DySO | PSNR: 22.9 -> 26.0 | Helped |
| add latent control swapping instead of no swapping (ablation) | DyST · DySO | R^cam: 0.72 -> 0.06 | Helped |
| add latent control swapping instead of swapping with 50% probability (ablation) | DyST · DySO | R^dyn: 1.07 -> 0.42 | Helped |
| add latent control swapping instead of averaging target and swapped-view latents (ablation) | DyST · DySO | R^dyn: 0.96 -> 0.42 | Helped |
Surprising findings
- Without latent control swap, camera and dynamics latents remain entangled (R_dyn = 1.26), so the swap scheme is needed for separation.
Limitations
- Object appearance is often blurry since L2 loss and only 3 input views limit detail on dynamic objects.
- Evaluated on simple scenes with one moving object and short, limited camera trajectories.
- Model has not converged after 4M training steps.
Experimental setup
Datasets: DySO Something-Something v2 MultiShapeNet ShapeNet
Benchmarks: DySO Something-Something v2
Metrics: PSNR LPIPS contrastiveness R_cam contrastiveness R_dyn
Baselines: No swap 50% swap Latent averaging
Builds on: RUST SRT Kubric MultiShapeNet
Closest papers by meaning
- OSN: Infinite Representations of Dynamic 3D Scenes from Monocular Videos (ICML 2024 · similarity 0.88)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.88)
- Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos (NeurIPS 2025 · similarity 0.88)
- DynaTokens: Teaching Dynamics to Camera-Controlled Video Models at Test Time (NeurIPS 2026 · similarity 0.88)
- DyBluRF: Dynamic Neural Radiance Fields from Blurry Monocular Video (CVPR 2024 · similarity 0.86)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.