AI Paper Atlas

Research map / Video / Video segmentation and tracking

Video Prediction Models as Rewards for Reinforcement Learning

Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, Pieter Abbeel

NeurIPS 2023 · Video › Video segmentation and tracking · approach: Stochastic video prediction · 11 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add exploration objective (beta from 0 to 1, Plan2Explore reward) to the video-likelihood reward (ablation)VIPER (VideoGPT-like backbone) · DMCtask returnImproved
replace VideoGPT-like video model with MaskGIT (ablation)VIPER · DMCtask returnHurt
replace MaskGIT with MaskGIT compute cost: up to 8 forward passes per frame instead of 1 (ablation)VIPER · DMCreward computation / RL training slowdown: 8x slowdown in reward computation, 2.5x in overall RL trainingHurt
replace VideoGPT with recurrent BYOL implicit-density video model (ablation)VIPER · DMCtask returnHurt
use BYOL instead of MaskGIT as the video model (ablation)VIPER · DMCtask returnImproved
increase video model context length k (ablation)VIPER (VideoGPT) · DMCtask returnImproved
context length k=0 (unconditional frame likelihood) (ablation)VIPER · DMCtask behaviorHurt
mask the scoreboard when training the video model (ablation)VIPER · Atari PongRL performance / stabilityImproved
train video model at reduced frame rate (subsample by factor 4) and stride likelihood contextVIPER · RLBenchtask successImproved
use VIPER rewards instead of task oracle sparse rewardsDreamerV3 · RLBenchtask successImproved
use VIPER rewards instead of task oracle dense rewards with stateDrQ / DreamerV3 · DMCtask returnHurt
use VIPER instead of AMP adversarial reward (single-task and multi-task)DrQ / DreamerV3 · DMC, Ataritask returnImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy