Research map / Video / Video segmentation and tracking
Moving Off-the-Grid: Scene-Grounded Video Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Moving Off-the-Grid (MooG), a recurrent transformer video model that learns scene-bound latent tokens via next-frame prediction, and shows its frozen and end-to-end readouts beat on-the-grid baselines (DINO, VideoMAE v2, grid recurrent) on point tracking, depth, and box tracking.
Problem
Grid-based vision representations tie tokens to fixed image locations, so content that moves across the image plane is not consistently represented by the same tokens, hurting downstream tasks requiring object or point tracking and geometry.
Approach
Decouple representation structure from image grid: a set of 1024 latent tokens is predicted by a self-attention predictor, corrected by cross-attending to new frame features, and decoded to pixels via cross-attention with coordinate queries; trained only with next-frame L2 prediction so tokens bind to and track scene elements.
Key results
- Frozen readouts: MooG reaches 0.839 AJ on MOVi-E points vs 0.822 for VideoMAE v2 (G) and 0.544 for DINOv2 (B); 0.687 AJ on DAVIS points.
- End-to-end, MooG obtains 0.094 AbsRel on Waymo depth, outperforming DPT ViT-L/E/22b baselines (0.154-0.161).
- On MOVi-E, MooG with 1024 tokens reaches PSNR 25.64 next-frame prediction after 500K steps; Table 4 shows MooG rivals TAPIR on 8-frame DAVIS (0.824 vs 0.823 AJ).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MooG instead of the strongest frozen grid-based baseline | Frozen readouts · MOVi-E point tracking | AJ: 0.778 -> 0.839 | Helped |
| Use MooG instead of the strongest frozen baseline | Frozen readouts · MOVi-E depth estimation | AbsRel: 0.0311 -> 0.0359 | Hurt |
| Use MooG instead of the strongest frozen baseline | Frozen readouts · MOVi-E object tracking | IoU: 0.793 -> 0.793 | No clear effect |
| Use MooG instead of the strongest frozen baseline | Frozen readouts · DAVIS point tracking | AJ: 0.720 -> 0.687 | Hurt |
| Use MooG instead of the strongest frozen baseline | Frozen readouts · Waymo object tracking | IoU: 0.708 -> 0.730 | Helped |
| Use MooG instead of the strongest end-to-end grid baseline | End-to-end readouts · MOVi-E point tracking | AJ: 0.902 -> 0.886 | Hurt |
| Use MooG instead of the strongest end-to-end grid baseline | End-to-end readouts · MOVi-E depth estimation | AbsRel: 0.0233 -> 0.0263 | Hurt |
| Use MooG instead of the strongest end-to-end grid baseline | End-to-end readouts · MOVi-E object tracking | IoU: 0.806 -> 0.803 | Hurt |
| Use MooG instead of the strongest end-to-end grid baseline | End-to-end readouts · DAVIS point tracking | AJ: 0.779 -> 0.778 | No clear effect |
| Use MooG instead of the strongest end-to-end grid baseline | End-to-end readouts · Waymo object tracking | IoU: 0.675 -> 0.719 | Helped |
| Use MooG instead of TAPIR | End-to-end point tracking · DAVIS-8 | AJ: 0.823 -> 0.824 | No clear effect |
| Use MooG instead of TAPIR | End-to-end point tracking · DAVIS-full | AJ: 0.580 -> 0.510 | Hurt |
Surprising findings
- In the end-to-end setting, the grid recurrent baseline (Grid Rec.) matches or beats MooG on MOVi-E points and depth, narrowing the advantage claimed for OTG tokens.
- Frozen VideoMAE v2 ViT-G matches or beats MooG on DAVIS points (0.720 vs 0.687) and on depth AbsRel (0.0311 vs 0.0359).
Limitations
- Evaluated mainly on low- and mid-level readouts; semantic tasks such as segmentation or classification are not tested and MooG struggles on semantic downstream tasks.
- Deterministic model with short prediction horizon; blurs under fast motion and may latch onto new elements when content appears or disappears.
- Scaling behaviour with model size and pretraining data not investigated.
Experimental setup
Datasets: Kubric MOVi-E Ego4D Kinetics700 Waymo Open Dataset TAP-Vid DAVIS
Benchmarks: MOVi-E DAVIS (TAP-Vid) Waymo Open
Metrics: AJ AbsRel IoU PSNR
Baselines: Grid Grid Rec. DINOv1 DINOv2 VideoMAE v2 TAP-Net TAPIR DPT SAVi++
Builds on: Perceiver Slot Attention PerceiverIO DINO VideoMAE v2
Closest papers by meaning
- LASER: A Neuro-Symbolic Framework for Learning Spatio-Temporal Scene Graphs with Weak Supervision (ICLR 2025 · similarity 0.86)
- EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision (ICLR 2024 · similarity 0.86)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.85)
- Visual Representation Learning with Stochastic Frame Prediction (ICML 2024 · similarity 0.85)
- GMOS: Grounding Moving Object Segmentation in 3D Space and Time (NeurIPS 2026 · similarity 0.85)
- Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.