Research map / Video / Video-language understanding
VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIBE, an annotation-free, training-free way to score and select VLM video summaries using information-bottleneck-based grounding (PMI between video and summary) and utility (PMI between summary and task) scores, validated by user studies with 243 participants.
Problem
Video summaries from VLMs are verbose and hurt human decision-making, while existing video caption evaluation relies on costly human-written references and ignores downstream task utility.
Approach
Approximates pointwise mutual information between video and summary (grounding) and between summary and task (utility) using VLM next-token probabilities with masked keywords (tf-idf), then ranks randomly sampled VLM outputs by a weighted sum of these scores via rejection sampling.
Key results
- Max-U summaries improved human accuracy over Naive VLM summaries on LongVideoBench (65.00% vs 46.43%) and LearningPaper24 (37.89% vs 28.42%), with significant gains.
- Max-U reduced response time versus Video Only on LearningPaper24 (46.69s vs 192.73s) and improved IES (1.28 vs 9.85).
- Utility score correlated with human accuracy strongly on LongVideoBench (Spearman r_s = 0.684, p < .001).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select summaries by utility score instead of randomly sampled VLM summaries | VIBE · LearningPaper24 | accuracy: 28.42 ± 12.68 -> 37.89 ± 10.04 | Helped |
| Select summaries by grounding score instead of randomly sampled VLM summaries | VIBE · LearningPaper24 | accuracy: 28.42 ± 12.68 -> 35.00 ± 7.91 | Helped |
| Select summaries by utility score instead of randomly sampled VLM summaries | VIBE · LongVideoBench | accuracy: 46.43 ± 13.94 -> 65.00 ± 15.47 | Helped |
| Select summaries by grounding score instead of randomly sampled VLM summaries | VIBE · LongVideoBench | accuracy: 46.43 ± 13.94 -> 59.23 ± 9.17 | Helped |
| Select summaries by utility score instead of randomly sampled VLM summaries | VIBE · SUTD-TrafficQA | accuracy: 76.96 ± 6.89 -> 84.81 ± 4.65 | Helped |
| Select summaries by grounding score instead of randomly sampled VLM summaries | VIBE · SUTD-TrafficQA | accuracy: 76.96 ± 6.89 -> 80.47 ± 3.21 | Helped |
| Select summaries by utility score instead of watching the video | VIBE · LearningPaper24 | response time: 192.73 ± 108.01 -> 46.69 ± 25.06 | Helped |
| Select summaries by grounding score instead of watching the video | VIBE · LearningPaper24 | response time: 192.73 ± 108.01 -> 61.13 ± 39.33 | Helped |
| Select summaries by utility score instead of watching the video | VIBE · LongVideoBench | response time: 202.35 ± 87.26 -> 65.86 ± 41.34 | Helped |
| Select summaries by grounding score instead of watching the video | VIBE · LongVideoBench | response time: 202.35 ± 87.26 -> 71.93 ± 35.96 | Helped |
| Select summaries by utility score instead of watching the video | VIBE · SUTD-TrafficQA | response time: 48.63 ± 21.30 -> 137.27 ± 81.64 | Hurt |
| Select summaries by grounding score instead of watching the video | VIBE · SUTD-TrafficQA | response time: 48.63 ± 21.30 -> 80.97 ± 41.11 | Hurt |
Surprising findings
- On SUTD-TrafficQA, Max-U summaries increased accuracy but response time rose (137.27s vs 48.63s for Video Only), so text summaries gave no efficiency benefit on short clips.
- Longer summaries had lower response time per word, suggesting shallower reading per word.
Limitations
- Utility score requires task labels, limiting applicability; grounding-only variant gives smaller gains.
- PMI estimates depend on tf-idf masking hyperparameters and VLM conditional probability biases.
- Multiple VLM inferences needed for grounding score; study uses between-subjects samples of about 10 stimuli per condition.
Experimental setup
Datasets: LearningPaper24 LongVideoBench SUTD-TrafficQA
Benchmarks: LearningPaper24 SUTD-TrafficQA LongVideoBench
Metrics: accuracy response time inverse efficiency score Spearman correlation t-test with Cohen's d
Baselines: Naive VLM summary Video Only Chain-of-Thought (CoT) summary author-written TL;DR OpenReview abstract
Builds on: information bottleneck pointwise mutual information tf-idf keyword masking
Closest papers by meaning
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.86)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.86)
- UVE: Are MLLMs Unified Evaluators for AI-Generated Videos? (NeurIPS 2025 · similarity 0.86)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.85)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.85)
- VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.