Research map / Video / Video generation
GenRec: Unifying Video Generation and Recognition with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
GenRec fine-tunes Stable Video Diffusion with random-frame latent masking and joint EDM denoising plus classification losses, so one model performs video recognition and frame-conditioned or class-conditioned video generation.
Problem
Video diffusion models learn strong spatial-temporal priors, but their generation training (corrupted inputs, single-frame conditioning) mismatches recognition (clean multi-frame inputs), so it is unclear whether they can be used for recognition while keeping generation and remaining robust when few
Approach
Condition the video diffusion model on a random subset of frames (latent masking) instead of only the first frame, train it jointly with noise-prediction and cross-entropy losses on intermediate UNet features, and use the classifier for class-guided sampling.
Key results
- GenRec reaches 75.8% top-1 on SSV2 and 87.2% on K400, and 46.5 / 49.3 FVD on SSV2 / EK-100 for class-conditioned image-to-video generation (12 frames).
- With only the first half of the video visible (rho=0.5), GenRec gets 57.7% on SSV2 early action prediction, retaining 76.6% of full-video accuracy.
- Removing generation supervision ('w/o G') lowers accuracy by 2.3 points at rho=0.3 on SSV2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add joint generation and recognition training (ablation) | GenRec · SSV2 early action prediction, ρ=0.3 | accuracy: 39.6 -> 41.9 | Helped |
| Add joint generation and recognition training (ablation) | GenRec · SSV2 limited interpolation, 2 fs | accuracy: 53.5 -> 55.7 | Helped |
| Use Up Index 2 rather than Up Index 1 for recognition features (ablation) | GenRec · SSV2 | accuracy: 71.8 -> 75.8 | Helped |
| Use Up Index 2 rather than Up Index 3 for recognition features (ablation) | GenRec · SSV2 | accuracy: 75.2 -> 75.8 | Helped |
| Set expected masking ratio to 50% rather than 75% (ablation) | GenRec · SSV2 | accuracy: +0.3 | Helped |
| Set expected masking ratio to 50% rather than 75% (ablation) | GenRec · SSV2 | FVD: +0.5 | Hurt |
| Set expected masking ratio to 87.5% rather than 75% (ablation) | GenRec · SSV2 | accuracy: -0.9 | Hurt |
| Set expected masking ratio to 87.5% rather than 75% (ablation) | GenRec · SSV2 | FVD: -0.3 | Helped |
| Set generation-loss balance ratio to 0 rather than 10 (ablation) | GenRec · SSV2 | FVD: 46.5 -> 1579.2 | Hurt |
| Vary the generation-loss balance ratio from 10 to 0, 1, 5, or 20 (ablation) | GenRec · SSV2 | accuracy: 75.8; 75.6; 75.5; 75.6; 75.5 | No clear effect |
| Add classification supervision to SVD fine-tuning | GenRec · SSV2 frame prediction | FVD: 55.5 -> 55.3 | No clear effect |
| Add classification supervision and classifier guidance | GenRec · SSV2 class-conditioned frame prediction | FVD: 50.3 -> 46.5 | Helped |
Surprising findings
- Recognition accuracy depends strongly on which frames are visible: uniformly spread 2 frames give 55.7% versus 28.9% for a 2-frame prefix.
Limitations
- Relies on fine-tuning a large pretrained video diffusion model with 2.1B total parameters and more pretraining data than prior recognition methods.
- Does not match the best multimodal-aligned recognition models (InternVideo2, OmniVec-2) on K400 and SSV2.
- Generation comparison to SEER uses different frame counts (16 vs 12) and SEER results from its official website.
Experimental setup
Datasets: Something-Something V2 Kinetics-400 UCF-101 Epic-Kitchen-100
Benchmarks: Something-Something V2 Kinetics-400 EK-100 UCF-101
Metrics: top-1 accuracy FVD accuracy retention ratio
Baselines: VideoMAE-L VideoMAE-H OmniMAE-H MVD-H Hiera-L Hiera-H MaskFeat-L InternVideo InternVideo2 OmniVec OmniVec-2 TATS
Builds on: Stable Video Diffusion EDM MAE classifier guidance
Closest papers by meaning
- LLM-grounded Video Diffusion Models (ICLR 2024 · similarity 0.88)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.87)
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.87)
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.87)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.87)
- Video Prediction by Modeling Videos as Continuous Multi-Dimensional Processes (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.