Research map / Video / Video-language understanding
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, Tamara Lee Berg, Mohit Bansal and 3 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add subtitle channel to video-only input (ablation) | HERO · VALUE val | Meta-Ave: 39.67 -> 52.52 | Improved |
| add video channel to subtitle-only input (ablation) | HERO · VALUE val | Meta-Ave: 40.40 -> 52.52 | Improved |
| HERO cross-modal fusion instead of two-stream late fusion (ablation) | HERO · VALUE val | Meta-Ave: 48.66 -> 52.52 | Improved |
| HERO fusion instead of sequence concat (ablation) | HERO · VALUE val | Meta-Ave: 51.53 -> 52.52 | Mixed |
| CLIP-ViT 2D features instead of ResNet (ablation) | HERO finetune-only · VALUE val | Meta-Ave: 48.29 -> 53.40 | Improved |
| CLIP-ViT+SlowFast instead of ResNet+SlowFast (ablation) | HERO finetune-only · VALUE val | Meta-Ave: 52.52 -> 54.52 | Improved |
| add ResNet to S3D features (ablation) | HERO finetune-only · VALUE val | Meta-Ave: 52.78 -> 49.34 | Hurt |
| multi-task by task type vs single-task | HERO finetune-only · VALUE test | Meta-Ave: 50.44 -> 52.00 | Improved |
| multi-task by domain vs single-task | HERO finetune-only · VALUE test | Meta-Ave: 50.44 -> 51.97 | Improved |
| all-task training vs single-task | HERO finetune-only · VALUE test | Meta-Ave: 50.44 -> 52.33 | Improved |
| finetune each task after all-task training | HERO finetune-only · VALUE test | Meta-Ave: 52.33 -> 52.89 | Improved |
| HERO pre-training before finetuning (single-task) | HERO · VALUE test | Meta-Ave: 50.44 -> 57.00 | Improved |
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.90)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.89)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
