Research map / Video / Video generation
Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a LLaMA-style autoregressive Transformer on VQ-tokenized video clips with next-token prediction, showing zero-shot video imitation from in-context demonstration videos (VidIT).
Problem
Enabling models to imitate semantics from a demonstration video and apply it to a new query scene without supervised demonstration-target pairs; prior in-context vision work relies on supervised image pairs and does not handle video semantics.
Approach
Train an autoregressive Transformer on continuous video token sequences with plain next-token prediction; at inference, prepend a demonstration video clip before the query so the model generates continuations that mimic the demonstrated semantics, emerging zero-shot without demonstration-query training pairs.
Key results
- In-class demonstrations raise P-Acc to 36.7 vs 29.6 with no demonstration for the Pretrain model (1.1B).
- VidIT achieves P-Acc 38.5 and PSNR 13.07 versus DeLVM 32.8 and LVM 27.1 on SSv2 automatic metrics.
- Human evaluation gives VidIT alignment 3.73 and quality 4.12 compared with LVM's 2.71 and 3.50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the baseline model with VidIT | LVM, DeLVM, and LWM · Something-Something v2 evaluation | P-Acc: 38.5 vs 32.8 | Helped |
| use an in-class rather than no demonstration (ablation) | VidIT Pretrain · SSv2 evaluation | P-Acc: 29.6 -> 36.7 | Helped |
| use an in-class rather than random demonstration (ablation) | VidIT Pretrain · SSv2 evaluation | V-Acc: 22.7 -> 24.7 | Helped |
| use an unrelated random demonstration rather than no demonstration (ablation) | VidIT Pretrain · SSv2 evaluation | V-Acc: 22.9 -> 22.7 | Hurt |
| fine-tune on in-domain SSv2 videos (ablation) | VidIT Pretrain · SSv2 evaluation, in-class demonstrations | P-Acc: 40.7 -> 48.8 | Helped |
| fine-tune with explicit imitation-format examples instead of continuous-video examples (ablation) | VidIT Pretrain · SSv2 evaluation, in-class demonstrations | P-Acc: 48.8 -> 40.7 | Hurt |
| add one 4-frame demonstration (ablation) | VidIT · SSv2 evaluation | P-Acc: 29.6 -> 35.6 | Helped |
| use two 4-frame demonstrations rather than one 4-frame demonstration (ablation) | VidIT · SSv2 evaluation | V-Acc: 22.9 -> 24.2 | Helped |
| extend one demonstration from 4 to 8 frames (ablation) | VidIT · SSv2 evaluation | P-Acc: 35.6 -> 36.7 | Helped |
| add text annotations during fine-tuning (ablation) | VidIT 300M Pretrain · SSv2 evaluation | V-Acc: 24.7 -> 28.8 | Helped |
Surprising findings
- Explicit imitation-format fine-tuning does not significantly improve over the zero-shot demonstration setting.
- Random demonstrations slightly reduce semantic accuracy relative to no demonstration.
Limitations
- Evaluation on longer video sequences is not thoroughly studied.
- Performance degrades when demonstrations are noisy or semantically unrelated to the query.
- Uses a frame-level image tokenizer without temporal compression.
Experimental setup
Datasets: Ego4d Kinetics-600 WebVid Something-Something v2 RT-1 MineRL VIPSeg
Benchmarks: Something-Something v2 RT-1 VIPSeg VP2 MineRL
Metrics: V-Acc P-Acc PSNR LPIPS FID FVD human quality/alignment/control scores
Baselines: LVM DeLVM LWM
Builds on: LVM LLaMA VQ-GAN
Closest papers by meaning
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.87)
- Autoregressive Video Generation without Vector Quantization (ICLR 2025 · similarity 0.87)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.87)
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (NeurIPS 2025 · similarity 0.86)
- Taming Teacher Forcing for Masked Autoregressive Video Generation (CVPR 2025 · similarity 0.86)
- VideoPoet: A Large Language Model for Zero-Shot Video Generation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.