Research map / Video / Video segmentation and tracking
Continuous Space-Time Video Resampling with Invertible Motion Steganography
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an invertible motion steganography module and a 3D implicit modulation sampler for continuous space-time video resampling, enabling motion-preserving temporal downsampling and non-integer frame rate conversion.
Problem
Video resampling must jointly downsample and upsample in space and time while preserving motion information lost during temporal downsampling without blur, and supporting flexible (including non-integer) spatial and temporal factors.
Approach
An invertible flow-based module (IMSM) hides high-frame-rate motion inside visually near-lossless low-frame-rate frames, paired with a 3D local implicit modulation sampler that maps features to arbitrary continuous space-time scales.
Key results
- Time 2x / Space 4x on Vimeo-90K reaches 37.78 dB RGB PSNR, about 3 dB above STAA (34.53 dB).
- Time 2x / Space 1x on Vid4 reaches 43.75 dB RGB PSNR versus 39.78 dB for the best prior STAA result.
- On SPMCS non-integer scales (S=2.0 to 4.0), the method outperforms VideoINR, MoTIF and MsMr at every scale, e.g. 35.74 dB vs 34.11 dB at S=2.0 in RGB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add invertible motion steganography (IMSM) (ablation) | video resampling model · Vimeo90k, T×2 S×4 | PSNR-Y: 38.86 -> 40.57 | Helped |
| add invertible motion steganography (IMSM) (ablation) | video resampling model · Vid4, T×2 S×4 | PSNR-Y: 30.61 -> 32.76 | Helped |
| add invertible motion steganography (IMSM) (ablation) | video resampling model · Vimeo90k, T×2 S×1 | PSNR-Y: 43.76 -> 50.52 | Helped |
| add invertible motion steganography (IMSM) (ablation) | video resampling model · Vid4, T×2 S×1 | PSNR-Y: 44.45 -> 46.77 | Helped |
| increase the upsampling backbone from 12 to 24 blocks (ablation) | video resampling model · Vimeo90k, T×2 S×4 | PSNR-Y: 39.52 -> 40.57 | Helped |
| increase the upsampling backbone from 12 to 24 blocks (ablation) | video resampling model · Vid4, T×2 S×4 | PSNR-Y: 31.55 -> 32.76 | Helped |
| increase the upsampling backbone from 12 to 24 blocks (ablation) | video resampling model · Vimeo90k, T×2 S×1 | PSNR-Y: 49.42 -> 50.58 | Helped |
| increase the upsampling backbone from 12 to 24 blocks (ablation) | video resampling model · Vid4, T×2 S×1 | PSNR-Y: 42.63 -> 46.57 | Helped |
| use the proposed continuous spatiotemporal resampling method instead of the strongest listed baseline | video resampling on SPMCS · SPMCS, S=2.0 | PSNR (RGB): 34.11 -> 35.74 | Helped |
| use the proposed continuous spatiotemporal resampling method instead of the strongest listed baseline | video resampling on SPMCS · SPMCS, S=2.4 | PSNR (RGB): 32.62 -> 35.03 | Helped |
| use the proposed continuous spatiotemporal resampling method instead of the strongest listed baseline | video resampling on SPMCS · SPMCS, S=2.8 | PSNR (RGB): 31.37 -> 34.17 | Helped |
| use the proposed continuous spatiotemporal resampling method instead of the strongest listed baseline | video resampling on SPMCS · SPMCS, S=3.2 | PSNR (RGB): 30.37 -> 33.36 | Helped |
Limitations
- Non-integer temporal motion information may be dispersed across multiple downsampled frames, which is not explicitly controlled.
- Quantization of stored RGB images makes the invertible mapping not perfectly reversible, requiring approximation.
Experimental setup
Datasets: Vimeo-90K Adobe240FPS
Benchmarks: Vimeo-90K Vid4 SPMCS
Metrics: PSNR SSIM NIQE
Baselines: XVFI FLAVR EDSC IFRNet RIFE EMA-VFI VFIFormer STAA ZSM STDAN MoTIF RSTT
Builds on: STAA BasicVSR++ Charbonnier loss enhanced affine coupling (invertible flow) Swin Transformer
Closest papers by meaning
- A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing (CVPR 2024 · similarity 0.84)
- MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation (CVPR 2025 · similarity 0.83)
- Enhancing Video Super-Resolution via Implicit Resampling-based Alignment (CVPR 2024 · similarity 0.83)
- Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction (NeurIPS 2025 · similarity 0.83)
- MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion (ICLR 2025 · similarity 0.83)
- 4DGC: Rate-Aware 4D Gaussian Compression for Efficient Streamable Free-Viewpoint Video (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.