Research map / Video / Video generation
SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SlowFast-VGen, a dual-speed action-driven long video generation system pairing a masked conditional video diffusion model (slow learning) with an inference-time Temp-LoRA fast learning module and a slow-fast learning loop, evaluated on video generation and long-horizon planning.
Problem
Action-driven video generation models produce short clips and lose consistency across temporally distant frames beyond the context window, since they lack a mechanism for storing episodic memory.
Approach
Stores episodic memory at inference time in a temporary LoRA module updated with a local learning rule on concatenated input-output latent chunks (action-agnostic, noise-augmented), and integrates this fast loop into outer slow learning over multi-episode data.
Key results
- Achieves FVD 514 versus 782 for the best baseline (AnimateDiff) and 1763 for Runway Gen-3 Turbo, with PSNR 19.21 and SSIM 60.53.
- Averages 0.37 scene cuts per long video versus 0.89 for Streaming-T2V, and SRC 93.71 versus 91.02.
- On Minecraft long-horizon planning, reaches waypoint distance 1.51 and FVD 446 versus 2.18 and 497 for StreamingT2V.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Temp-LoRA for episodic memory (ablation) | SlowFast-VGen · Long video generation | SCuts: 1.88 -> 0.37 | Helped |
| Add Temp-LoRA for episodic memory (ablation) | SlowFast-VGen · Long video generation | SRC: 89.04 -> 93.71 | Helped |
| Use previous video chunk rather than only a single-frame image as input (ablation) | SlowFast-VGen · Long video generation | SCuts: 1.24 -> 0.37 | Helped |
| Use previous video chunk rather than only a single-frame image as input (ablation) | SlowFast-VGen · Long video generation | SRC: 90.01 -> 93.71 | Helped |
| Use the local learning rule rather than training Temp-LoRA on sampled full sequences (ablation) | SlowFast-VGen · Long video generation | SRC: 90.27 -> 93.71 | Helped |
| Use the local learning rule rather than training Temp-LoRA on sampled full sequences (ablation) | SlowFast-VGen · Long video generation | SCuts: 0.36 -> 0.37 | No clear effect |
| Use the video-specific Temp-LoRA rather than the original Temp-LoRA structure (ablation) | SlowFast-VGen · Long video generation | SRC: 92.24 -> 93.71 | Helped |
| Add the slow-fast learning loop (ablation) | SlowFast-VGen · RLBench | Dist: 0.055 -> 0.013 | Helped |
| Add the slow-fast learning loop (ablation) | SlowFast-VGen · Minecraft | Dist: 2.23 -> 1.51 | Helped |
| Use SlowFast-VGen instead of the strongest listed baseline | Video generation test set · Video generation test set | FVD: 782 -> 514 | Helped |
| Use SlowFast-VGen instead of the strongest listed baseline | Video generation test set · Video generation test set | PSNR: 18.04 -> 19.21 | Helped |
| Use SlowFast-VGen instead of the strongest listed baseline | Video generation test set · Video generation test set | LPIPS: 20.65 -> 25.06 | Hurt |
Surprising findings
- Temp-LoRA adds only about 7% inference time (12.93s to 13.81s per sample) and roughly 350 MB memory.
- Training Temp-LoRA on the full sampled sequence instead of local input-output pairs gives fewer scene cuts (0.36) but over-smooths and blurs later frames.
Limitations
- Model cannot serve as a general world model because many scenarios are not covered by the dataset.
- Inference-time fast learning adds computation overhead.
- Slow-fast learning loop is intensive for full pre-training, so it is mainly applied to finetuning.
Experimental setup
Datasets: EPIC-KITCHENS OpenX-Embodiment Metaworld RLBench HRI Driving Dataset (HDD) Unreal Engine collected data Minecraft recordings
Benchmarks: FVD PSNR SSIM LPIPS SCuts SRC RLBench long-horizon planning Minecraft navigation planning
Metrics: FVD PSNR SSIM LPIPS SCuts SRC human evaluation score waypoint distance
Baselines: AVDC Streaming-T2V Runway Gen-3 Turbo AnimateDiff SEINE iVideoGPT
Builds on: ModelScopeT2V Temp-LoRA LoRA AVDC iVideoGPT
Closest papers by meaning
- FastLightGen: Fast and Light Video Generation with Fewer Steps and Parameters (CVPR 2026 · similarity 0.88)
- VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis (ICLR 2025 · similarity 0.88)
- Learning World Models for Interactive Video Generation (NeurIPS 2025 · similarity 0.87)
- WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception (NeurIPS 2025 · similarity 0.87)
- LVD-2M: A Long-take Video Dataset with Temporally Dense Captions (NeurIPS 2024 · similarity 0.87)
- FastVID: Dynamic Density Pruning for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.