Research map / Video / Video-language understanding
E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces E.T. Bench, a 7.3K-sample open-ended event-level, time-sensitive video-LLM benchmark, plus the E.T. Chat model (timestamp prediction as embedding matching) and the E.T. Instruct 164K instruction-tuning dataset.
Problem
Existing video-LLM benchmarks rely on video-level, mostly multiple-choice QA and short clips, so they do not assess fine-grained event-level, time-sensitive understanding; current models also struggle with timestamp representation and multi-event training data.
Approach
Reformulate timestamp prediction as an embedding matching problem: the LLM emits a special <vid> token whose hidden state is matched by cosine similarity to per-frame video token embeddings, with a smoothed matching loss and bidirectional attention among video tokens.
Key results
- E.T. Chat achieves 44.6 accuracy on RAR and 62.5 F1 on VHD, leading open-source MLLMs on most E.T. Bench tasks.
- Gemini-1.5-Pro scores best on several referring and grounding tasks among evaluated models, e.g., 61.1 on RAR.
- Most open-source Video-LLMs struggle on grounding and dense captioning; e.g., Video-LLaMA-2 scores 0.0-0.1 F1 on several grounding tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench TVG | F1: 26.2 -> 38.6 | Helped |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench EPM | F1: 4.6 -> 10.2 | Helped |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench TAL | F1: 18.2 -> 30.8 | Helped |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench VHD | F1: 48.2 -> 62.5 | Helped |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench GVQ | recall: 2.2 -> 3.7 | Helped |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench EVC | accuracy: 40.8 -> 37.0 | Hurt |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench RVQ | accuracy: 40.2 -> 33.6 | Hurt |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench EVS | F1: 29.7 -> 25.4 | Hurt |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench DVC | F1: 40.2 -> 38.4 | Hurt |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench SLC | sentence similarity: 15.2 -> 14.6 | Hurt |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench TEM | recall: 18.0 -> 16.5 | Hurt |
| Use E.T. Chat with embedding-based timestamp matching and E.T. Instruct 164K | Video-LLMs · E.T. Bench DVC | sentence similarity: 18.6 -> 19.7 | Helped |
Surprising findings
- Some Image-LLMs (e.g., Qwen-VL-Chat, XComposer) match or beat many Video-LLMs on referring tasks, attributed to short referring videos.
- Some Video-LLMs (Video-LLaVA, Video-LLaMA-2) fail to output required timestamp formats on many tasks.
Limitations
- Image-LLM evaluation uses only 8 uniformly sampled frames, and commercial models are evaluated on a 470-sample subset.
- Some task formulations (TAL, EVS, VHD) were modified from prior work to suit LLMs, limiting direct comparability.
Experimental setup
Datasets: Perception Test Charades-STA STAR QVHighlights Ego4D-NLQ THUMOS'14 THUMOS'15 TVSum SumMe YouTube Highlights YouCook2 HiREST
Benchmarks: E.T. Bench
Metrics: accuracy F1 recall sentence similarity
Baselines: LLaVA-1.5 Video-LLaVA LLaMA-VID PLLaVA VTimeLLM VTG-LLM TimeChat LITA GPT-4V GPT-4o Gemini-1.5-Pro Gemini-1.5-Flash
Builds on: Q-Former VTimeLLM TimeChat LITA
Closest papers by meaning
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.91)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.90)
- CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding (ICLR 2025 · similarity 0.90)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.90)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.