Research map / Video / Video segmentation and tracking
Visual Representation Learning with Stochastic Frame Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RSP, which repurposes a stochastic video generation model (SVG-style posterior/prior with categorical latents) as a self-supervised image representation learner from Kinetics-400 videos, combined with an auxiliary MAE objective sharing the decoder.
Problem
Future frame prediction as a self-supervised objective for image representation learning struggles because future frames are under-determined (multiple plausible futures from one frame), and deterministic predictors learn weak representations from complex real-world videos.
Approach
Re-configure stochastic video generation (SVG) for representation learning: a learned time-dependent prior predicts the posterior latent of the future frame from the current frame; categorical latents (straight-through) are used; an auxiliary masked autoencoding objective shares the decoder via cross-attention on distinct inputs, enabling synergistic, parameter-efficient training.
Key results
- RSP achieves 35.6% success on RLBench versus 6.0% for SiamMAE, and 36.0% average in the main RLBench table compared with 13.5% for MAE (as stated in the text).
- On DAVIS video object segmentation, RSP reaches 60.1 J&F mean with ViT-S/16 versus 53.5 for MAE and 59.5 for DINO (Kinetics-400 pre-trained).
- Deterministic frame prediction achieves 54.4 J&F versus 60.1 for the stochastic version on DAVIS; the auxiliary MAE with shared decoder improves from 58.1 (separate decoder) to 60.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace deterministic frame prediction with stochastic frame prediction (ablation) | RSP · DAVIS | J&F_m: 54.4 -> 60.1 | Helped |
| use categorical rather than Gaussian latent variables (ablation) | RSP · DAVIS | J&F_m: 54.1 -> 60.1 | Helped |
| add the auxiliary MAE objective (ablation) | RSP · DAVIS | J&F_m: +2.4%p | Helped |
| use a shared rather than separate decoder (ablation) | RSP with MAE · DAVIS | J&F_m: 58.1 -> 60.1 | Helped |
| set KL scale to 0.1 rather than the default 0.01 (ablation) | RSP · DAVIS | J&F_m: 60.1 -> 56.1 | Hurt |
| apply the same augmentation to current and future frames (ablation) | RSP · DAVIS | J&F_m: 53.7 -> 60.1 | Helped |
| use mild Gaussian noise instead of no future-frame augmentation (ablation) | RSP · DAVIS | J&F_m: 58.3 -> 60.1 | Helped |
| apply masking to the future frame at scale 0.95 rather than the default noise augmentation (ablation) | RSP · DAVIS | J&F_m: 60.1 -> 55.8 | Hurt |
| use distinct rather than shared encoder-decoder projections for the objectives (ablation) | RSP · DAVIS | J&F_m: 56.6 -> 60.1 | Helped |
| concatenate latent variables and patch representations as tokens rather than along the channel dimension (ablation) | RSP · DAVIS | J&F_m: 54.1 -> 60.1 | Helped |
| pre-train RSP rather than Dino | ViT-S/16 representations · CortexBench MetaWorld | normalized score: 82.4 -> 84.5 | Helped |
| pre-train RSP rather than SiamMAE | ViT-S/16 representations · RLBench Saucepan | success rate (%): 22.5 -> 93.4 | Helped |
Surprising findings
- A deterministic frame predictor underperforms badly (54.4 vs 60.1 J&F), suggesting stochasticity is essential for learning from complex real-world videos.
- Gaussian latents underperform categorical latents substantially (54.1 vs 60.1 J&F on DAVIS).
Limitations
- Generated frame quality is not high, though fidelity is not the focus.
- Experiments limited by compute to ViT-S/16 with short training budgets and no large-scale models.
- Only single-frame-pair prediction is explored; extension to multiple frames is future work.
Experimental setup
Datasets: Kinetics-400 ImageNet CortexBench RLBench FrankaKitchen DAVIS VIP JHMDB
Benchmarks: CortexBench RLBench FrankaKitchen DAVIS VIP JHMDB
Metrics: normalized score success rate interquartile mean J&F mean J mean F mean mIoU PCK@0.1 PCK@0.2
Baselines: SimCLR MoCo v3 DINO MAE SiamMAE
Builds on: stochastic video generation (SVG) masked autoencoders SiamMAE VQ/categorical latent variational models (DreamerV2-style) straight-through estimator
Closest papers by meaning
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.88)
- Modeling Video As Stochastic Processes for Fine-Grained Video Representation Learning (CVPR 2023 · similarity 0.88)
- StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representation (CVPR 2026 · similarity 0.87)
- Token Bottleneck: One Token to Remember Dynamics (NeurIPS 2025 · similarity 0.87)
- STDiff: Spatio-Temporal Diffusion for Continuous Stochastic Video Prediction (AAAI 2024 · similarity 0.87)
- Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.