Research map / Video / Video-language understanding
OmniViD: A Generative Framework for Universal Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies video recognition, captioning, QA, dense captioning and object tracking as token generation by adding time and box tokens to a BART-based encoder-decoder with a Q-Former-style MQ-former.
Problem
Video understanding tasks (recognition, captioning, dense captioning, tracking) use task-specific heads and annotation formats, preventing a single shared model and output space.
Approach
Extend the language vocabulary with discretized time and box tokens so all video tasks become video-grounded token generation trained with cross-entropy, using an MQ-former with content, sentence and box queries to fuse prompts into visual features.
Key results
- Achieves 83.6% top-1 on Kinetics-400 and 71.3% on SSV2 for action recognition, beating VideoSwin-B by 0.9 and 1.7 points.
- Reaches 56.6 CIDEr on MSRVTT clip captioning and 42.3% accuracy on MSRVTT video QA.
- Obtains 79.6 normalized precision on LaSOT and 88.9 on TrackingNet; dense captioning SODA_c 5.6 on ActivityNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OmniViD instead of UniFormer-B | Video action recognition · Kinetics-400 | Top-1 accuracy: 82.9 -> 83.6 | Helped |
| Use OmniViD instead of UniFormer-B | Video action recognition · Something-Something V2 | Top-1 accuracy: 71.2 -> 71.3 | Helped |
| Use OmniViD instead of SwinBERT | Video captioning · MSRVTT | CIDEr: 53.8 -> 56.6 | Helped |
| Use OmniViD instead of SwinBERT | Video captioning · MSVD | CIDEr: 120.6 -> 122.5 | Helped |
| Use OmniViD instead of OmniVL | Video question answering · MSRVTT | accuracy: 44.1 -> 42.3 | Hurt |
| Use OmniViD instead of OmniVL | Video question answering · MSVD | accuracy: 51.0 -> 47.7 | Hurt |
| Use OmniViD instead of Vid2seq | Dense video captioning · ActivityNet Captions validation | CIDEr: 30.10 -> 26.00 | Hurt |
| Use OmniViD instead of Vid2seq | Dense video captioning · ActivityNet Captions validation | SODA c: 5.8 -> 5.6 | Hurt |
| Use OmniViD instead of UNINEXT | Visual object tracking · LaSOT | normalized precision: 80.7 -> 79.6 | Hurt |
| Use OmniViD instead of ARTrack | Visual object tracking · TrackingNet | normalized precision: 88.7 -> 88.9 | Helped |
| Add text queries (ablation) | OmniViD · VideoQA | accuracy: 40.4 -> 42.3 | Helped |
| Add text queries (ablation) | OmniViD · Visual object tracking | normalized precision: 79.2 -> 79.6 | Helped |
Surprising findings
- Joint training across tasks degrades spatial-temporal localization performance compared with separate per-task training.
Limitations
- Joint multi-task training degrades localization performance relative to separate training.
- Dense captioning recall is lower than traditional methods due to fixed localization head predictions being compared differently.
- Pretraining-free results trail large-scale pretrained Vid2Seq on dense captioning.
Experimental setup
Datasets: Kinetics-400 Something-Something V2 MSRVTT MSVD ActivityNet Captions TrackingNet LaSOT GOT10K
Benchmarks: Kinetics-400 Something-Something V2 MSRVTT MSVD ActivityNet Captions LaSOT TrackingNet HMDB-51 UCF101
Metrics: top-1 accuracy BLEU4 METEOR CIDEr SODA_c precision normalized precision success
Baselines: VideoSwin-B UniFormer-B SwinBERT Vid2seq PDVC SeqTrack ARTrack UNINEXT JustAsk OmniVL
Builds on: BLIP-2 Q-Former Vid2Seq BART VideoSwin Pix2Seq
Closest papers by meaning
- Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks (CVPR 2025 · similarity 0.89)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.88)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.88)
- VideoPrism: A Foundational Visual Encoder for Video Understanding (ICML 2024 · similarity 0.88)
- Omnipotent Distillation with LLMs for Weakly-Supervised Natural Language Video Localization: When Divergence Meets Consistency (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.