Research map / Video / Video generation
Seer: Language Instructed Video Prediction with Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Seer inflates a pretrained text-to-image latent diffusion model (Stable Diffusion v1.5) along time with frozen 2D layers, a windowed causal temporal attention, and a Frame Sequential Text decomposer to produce per-frame sub-instructions for text-conditioned video prediction.
Problem
Text-conditioned video prediction (TVP) for robot planning requires generating future frames from reference frames and a task-level instruction, with high fidelity, temporal coherence, and frame-level instruction alignment, while avoiding large text-video datasets and heavy compute.
Approach
Inflate a frozen pretrained T2I latent diffusion U-Net to 3D with lightweight scaled autoregressive window temporal attention, and decompose the global CLIP text embedding into per-frame sub-instruction embeddings via a Frame Sequential Text decomposer, initialized as identity, with frame-individual cross-attention.
Key results
- On SSv2, Seer reaches FVD 112.9 and KVD 0.12, versus 163.2 for VideoFusion (the best baseline), about a 31% FVD improvement.
- On BridgeData, Seer achieves FVD 246.3 and KVD 0.55; on Epic100, FVD 271.4.
- Human evaluation (54 evaluators, 99 clips) shows Seer preferred at least 72.2% over baselines in text-conditioned video manipulation; the abstract reports 83.7% average preference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Seer instead of VideoFusion | Text-conditioned video prediction · SSv2 | FVD: 163.2 -> 112.9 | Helped |
| Replace window autoregressive temporal attention with bi-directional temporal attention (ablation) | Seer · SSv2 | FVD: 112.9 -> 258.2 | Hurt |
| Replace window autoregressive temporal attention with directed temporal attention (ablation) | Seer · SSv2 | FVD: 112.9 -> 222.3 | Hurt |
| Replace window autoregressive temporal attention with autoregressive spatial-temporal attention (ablation) | Seer · SSv2 | FVD: 112.9 -> 200.1 | Hurt |
| Replace identity initialization with random initialization (ablation) | FSText in Seer · SSv2 | FVD: 112.9 -> 367.9 | Hurt |
| Remove FSText fine-tuning (ablation) | Seer · SSv2 | FVD: 112.9 -> 328.2 | Hurt |
| Also fine-tune spatial cross-attention (ablation) | Seer with FSText and temporal attention fine-tuned · SSv2 | FVD: 112.9 -> 1807 | Hurt |
| Reduce FSText depth from 8 layers to 2 (ablation) | FSText in Seer · SSv2 | FVD: 112.9 -> 238.6 | Hurt |
| Reduce FSText depth from 8 layers to 4 (ablation) | FSText in Seer · SSv2 | FVD: 112.9 -> 229.7 | Hurt |
| Remove temporal attention from FSText (ablation) | FSText in Seer · SSv2 | FVD: 112.9 -> 125.8 | Hurt |
| Remove text-sequential attention from FSText (ablation) | FSText in Seer · SSv2 | FVD: 112.9 -> 127.7 | Hurt |
| Unfreeze pretrained 2D spatial layers (ablation) | Seer · 16-frame training | training time (sec./iter.): 0.75 -> 0.96 | Hurt |
Surprising findings
- Random initialization of the FSText decomposer degrades FVD from 112.9 to 367.9, so identity initialization is essential.
- Fine-tuning the cross-attention layers together with temporal attention badly hurts performance (FVD 1807 on SSv2), suggesting the decomposer is tightly coupled to the frozen diffusion prior.
Limitations
- Performance on 16-frame videos degrades relative to the 12-frame model it was trained for.
- Zero-shot generalization to EGO4D and unseen objects is limited; the model struggles with interactions such as manipulating objects with hands.
- Not pretrained on video data, which limits quality on UCF-101 (FVD 260.7 vs Make-A-Video 81.25).
Experimental setup
Datasets: Something-Something V2 BridgeData EpicKitchens-100 EGO4D UCF-101
Benchmarks: Something-Something V2 BridgeData EpicKitchens-100 EGO4D UCF-101
Metrics: FVD KVD Inception score human preference
Baselines: Tune-A-Video MCVD PVDM VideoFusion TATS MAGE SimVP
Builds on: Stable Diffusion Tune-A-Video I3D classifier-free guidance DDIM
Closest papers by meaning
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.88)
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.88)
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.88)
- Video Language Planning (ICLR 2024 · similarity 0.87)
- VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning (ICLR 2025 · similarity 0.87)
- Solving New Tasks by Adapting Internet Video Knowledge (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.