Research map / Video / Video-language understanding
Aha! - Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Aha, a causal, autoregressive vision-language highlight detector with decoupled relevance/informativeness/uncertainty heads and a Dynamic SinkCache, trained on a new 22k-video HIHD dataset, reporting SOTA on TVSum and Mr.HiSum under online constraints.
Problem
Highlight detection for streaming video is mostly done offline with full-video access, so existing methods cannot support real-time, step-by-step, task-conditioned decisions under strict causal constraints.
Approach
A frozen SigLIP encoder and a LoRA-tuned Qwen-based decoder process frames causally; the hidden state feeds relevance, informativeness, uncertainty and LM heads, and a Dynamic SinkCache keeps the task-query tokens as the attention sink with a 2048-token sliding window for constant memory.
Key results
- On TVSum, zero-shot Aha reaches 91.6 top-5 mAP and domain-adapted Aha reaches 93.0 mAP, above TR-DETR's 87.1.
- On Mr.HiSum test, Aha reaches 64.19 mAP@50 and 32.66 mAP@15, versus PGL-SUM's 55.89 and 27.45.
- On Charades-STA streaming moment retrieval, Aha (w=8) reaches 50.7 R@0.5 and 27.9 R@0.7, above MMDuet (w=8) at 42.4 and 18.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Aha zero-shot instead of TR-DETR | TVSum · TVSum | top-5 mAP: 87.1 -> 91.6 | Helped |
| grid-search scoring parameters on the TVSum validation set | Aha · TVSum | top-5 mAP: 91.6 -> 93.0 | Helped |
| use Aha relevance-only scoring instead of PGL-SUM | Mr.HiSum test set · Mr.HiSum test set | mAP@50: 55.89 -> 64.19 | Helped |
| add the informativeness contribution (ablation) | Aha · TVSum | top-5 mAP: 83.2 -> 93.0 | Helped |
| add the relevance contribution (ablation) | Aha · TVSum | top-5 mAP: 77.3 -> 93.0 | Helped |
| add uncertainty scoring (ablation) | Aha · TVSum | top-5 mAP: 89.8 -> 93.0 | Helped |
| retain task language conditioning (ablation) | Aha · TVSum | top-5 mAP: 81.2 -> 93.0 | Helped |
| use Dynamic SinkCache instead of Standard SinkCache (ablation) | Aha · TVSum | top-5 mAP: 92.6 -> 93.0 | Helped |
| use Dynamic SinkCache instead of Unbounded KV Cache (ablation) | Aha · TVSum | top-5 mAP: 91.7 -> 93.0 | Helped |
| retain task-focused memory rather than only a sliding window (ablation) | Aha · TVSum | top-5 mAP: 69.5 -> 93.0 | Helped |
| retain task-focused memory rather than only an initial static window (ablation) | Aha · TVSum | top-5 mAP: 63.2 -> 93.0 | Helped |
| train with video quality dropout | Aha · TVSum under video corruptions | top-5 mAP: 93.0; 92.6 (Δ₀.₄); 91.2 (Δ₁.₈); 88.9 (Δ₄.₁); 88.2 (Δ₄.₈) | Mixed |
Surprising findings
- Removing language conditioning drops TVSum mAP from 93.0 to 81.2, while a standard SinkCache gives 92.6 and an unbounded KV cache 91.7.
- Aha underperforms MMDuet on the MAGQA in-span score and on YouCook2 dense captioning, with the paper noting repeated predictions.
Limitations
- Engagement-based (replay count) labels are an imperfect proxy for true highlight importance and may favor clickbait.
- Inference runs at 1 FPS, so it lags live streams without frame skipping.
- Uncertainty head is trained without ground-truth uncertainty labels, and the weighting is static.
Experimental setup
Datasets: HIHD Mr.HiSum Shot2Story COIN MMDuetIT
Benchmarks: TVSum Mr.HiSum Charades-STA YouCook2 MAGQA
Metrics: top-5 mAP Kendall tau Spearman rho mAP@50 mAP@15 R@0.5 R@0.7 F1 SODA_c CIDEr
Baselines: TR-DETR QD-DETR UniVTG PGL-SUM LLMVS SL-module iPTNet DSNet MMDuet VTG-LLM TimeChat
Builds on: StreamingLLM SinkCache MMDuet LLaVA-OneVision Mr.HiSum
Closest papers by meaning
- ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks (NeurIPS 2025 · similarity 0.85)
- Streaming Detection of Queried Event Start (NeurIPS 2024 · similarity 0.85)
- ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries (AAAI 2025 · similarity 0.85)
- Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.85)
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.84)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.