Research map / Video / Video generation
Learning Conditional Space-Time Prompt Distributions for Video Class-Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoSTEP, which models a conditional space-time prompt distribution with a diffusion model over frozen CLIP ViT-B/16 features for video class-incremental learning, and uses a frame-grid to get temporal modeling from the image transformer.
Problem
Video class-incremental learning suffers from catastrophic forgetting; existing deterministic prompt pools or prompt generators capture video diversity poorly and generalize weakly to future tasks and classes.
Approach
Model the prompt space as a distribution p(P|E) conditioned on the input video's ViT features, sampled via a conditional diffusion model (U-Net denoiser) during training and inference, combined with a frame-grid space-time prompting scheme that lets a frozen image ViT model inter-frame patch relations without extra parameters.
Key results
- CoSTEP reaches 86.95% average accuracy on UCF101 with 25 tasks and 61.84% on HMDB51 with 25 tasks, beating ST-Prompt (85.67, 60.54).
- On UCF101 with the vCLIMB split, CoSTEP gets 96.51% accuracy and 1.99 backward forgetting with 10 tasks, versus PIVOT at 93.36% and 4.47.
- Ablation: using only the diffusion loss gives 49.54% on UCF101, versus 96.51% with both losses combined.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use conditional prompt diffusion instead of ST-Prompt | CoSTEP · UCF101, 10 tasks | average accuracy: 85.54 -> 86.71 | Helped |
| use conditional prompt diffusion instead of PIVOT | CoSTEP · UCF101, 10 tasks | average accuracy: 93.36 -> 96.51 | Helped |
| use conditional prompt diffusion instead of PIVOT | CoSTEP · UCF101, 10 tasks | backward forgetting: 4.47 -> 1.99 | Helped |
| replace conditional prompt diffusion with a single sequentially tuned prompt set (ablation) | CoSTEP · UCF101, 10 tasks | average accuracy: 96.51 -> 74.98 | Hurt |
| replace conditional prompt diffusion with prompt-pool selection (L2P) (ablation) | CoSTEP · UCF101, 10 tasks | average accuracy: 96.51 -> 89.61 | Hurt |
| replace conditional prompt diffusion with CODA-Prompt (ablation) | CoSTEP · UCF101, 10 tasks | average accuracy: 96.51 -> 92.07 | Hurt |
| replace conditional prompt diffusion with DAP prompt generation (ablation) | CoSTEP · UCF101, 10 tasks | average accuracy: 96.51 -> 85.38 | Hurt |
| replace conditional prompt diffusion with BPT (ablation) | CoSTEP · UCF101, 10 tasks | average accuracy: 96.51 -> 93.93 | Hurt |
| replace conditional prompt diffusion with VAE (ablation) | CoSTEP · UCF101, 10 tasks | average accuracy: 96.51 -> 95.18 | Hurt |
| replace conditional prompt diffusion with GAN (ablation) | CoSTEP · UCF101, 10 tasks | average accuracy: 96.51 -> 92.32 | Hurt |
| sample prompts from N(0, I) rather than learn a prompt distribution (ablation) | CoSTEP · UCF101, 10 tasks | average accuracy: 96.51 -> 58.37 | Hurt |
| sample prompts from N(µ(P), Σ(P)) rather than learn a conditional diffusion distribution (ablation) | CoSTEP · UCF101, 10 tasks | average accuracy: 96.51 -> 75.92 | Hurt |
Surprising findings
- Sampling prompts from a simple Gaussian N(0, I) or a Gaussian fitted to prompt statistics performs poorly (58.37% and 75.92% on UCF101), far below the diffusion model.
Limitations
- Relies on storing a limited set of video feature vectors (20 per class) for replay, which is a memory cost.
- Diffusion sampling requires iterative denoising at inference time.
Experimental setup
Datasets: UCF101 HMDB51 Something-Something V2
Benchmarks: UCF101 HMDB51 Something-Something V2
Metrics: average accuracy backward forgetting
Baselines: iCaRL UCIR PODNet TCD FrameMaker L2P HCE S-Prompts ST-Prompt CODA-Prompt DAP PIVOT
Builds on: L2P DualPrompt CODA-Prompt VPT denoising diffusion probabilistic model ST-Prompt
Closest papers by meaning
- State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding (NeurIPS 2025 · similarity 0.88)
- STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding (CVPR 2025 · similarity 0.86)
- Optical-Flow Guided Prompt Optimization for Coherent Video Generation (CVPR 2025 · similarity 0.86)
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.85)
- LLM-grounded Video Diffusion Models (ICLR 2024 · similarity 0.85)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.