Research map / Video / Video-language understanding
Streaming Detection of Queried Event Start
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Streaming Detection of Queried Event Start (SDQES), a benchmark (EgoSDQES) built from Ego4D with LLM-generated streaming queries, plus streaming recall/minimum-distance metrics and adapter-based streaming baselines on frozen vision-language backbones.
Problem
Detecting the start of natural-language-described complex events in untrimmed, egocentric video streams with low latency; existing ODAS uses closed action vocabularies and offline language localization is not streaming.
Approach
Formulate SDQES with a streaming recall metric that considers only the first K predictions within an asymmetric tolerance window, and build streaming adapters (QRNN, ST-style conv, RetNet, MLP) on frozen image/video VLM backbones so each new frame costs constant compute.
Key results
- On 1-minute clips, EgoVLP + QR-Adapter reaches SR@1 of 28.8 and LaViLa + QR-Adapter 29.1 versus 16.9 for zero-shot CLIP.
- On 5-minute clips, EgoVideo + Adapter achieves SR@1 of 16.0 and SR@3 of 26.4, the best among reported models.
- Temporal adapters add roughly 13% to backbone FLOPs and about 5% to computation latency (RetNet is slower without an optimized CUDA kernel); a human expert baseline reaches SR@1 of 72.4 on 87 untrimmed annotations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add QR-Adapter instead of the non-temporal Adapter (ablation) | CLIP · EgoSDQES 1-minute clips | SR@1: 19.5 -> 23.7 | Helped |
| add QR-Adapter instead of the non-temporal Adapter (ablation) | LaViLa · EgoSDQES 1-minute clips | SR@1: 19.5 -> 29.1 | Helped |
| add QR-Adapter instead of the non-temporal Adapter (ablation) | EgoVLP · EgoSDQES 1-minute clips | SR@1: 18.1 -> 28.8 | Helped |
| add QR-Adapter instead of the non-temporal Adapter (ablation) | CLIP · EgoSDQES 5-minute clips | SR@1: 8.9 -> 9.1 | Helped |
| add QR-Adapter instead of the non-temporal Adapter (ablation) | LaViLa · EgoSDQES 5-minute clips | SR@1: 8.7 -> 9.3 | Helped |
| add QR-Adapter instead of the non-temporal Adapter (ablation) | EgoVLP · EgoSDQES 5-minute clips | SR@1: 8.4 -> 9.7 | Helped |
| train with EgoSDQES data using an Adapter | CLIP · EgoSDQES 1-minute clips | SR@1: 16.9 -> 19.5 | Helped |
| replace QR-Adapter with ST-Adapter (ablation) | EgoVLP · EgoSDQES 1-minute clips | SR@1: 28.8 -> 17.4 | Hurt |
| replace QR-Adapter with RN-Adapter (ablation) | EgoVLP · EgoSDQES 1-minute clips | SR@1: 28.8 -> 25.7 | Mixed |
| use a sliding window instead of streaming adapters (ablation) | EgoVLP · EgoSDQES 5-minute video efficiency | Multiply Adds: +298.5% | Hurt |
Surprising findings
- The 1D-convolution ST-Adapter performs poorly, with EgoVLP + ST-Adapter at 17.4 SR@1 on 1-minute clips, barely above the non-temporal adapter.
- Standard p-mAP is temporally invariant, so it cannot distinguish a model that alerts correctly on the first attempt from one that succeeds only after false positives.
Limitations
- Dataset inherits errors and omissions in Ego4D narrations and annotations.
- Start times are inherently ambiguous, so tolerance windows are needed.
- No error bars are reported because multiple seeds were infeasible due to long-sequence training cost.
Experimental setup
Datasets: Ego4D Moments Ego4D NLQ Ego4D Narrations EgoSDQES
Benchmarks: EgoSDQES
Metrics: Streaming Recall (SR@K) Streaming Minimum Distance (SMD@K) parameter count MACs FLOPs computation latency
Baselines: Zero-Shot CLIP CLIP + Adapter LaViLa + Adapter EgoVLP + Adapter EgoVideo + Adapter
Builds on: ST-Adapter AIM Quasi-Recurrent Neural Networks RetNet online detection of action start
Closest papers by meaning
- StreamForest: Efficient Online Video Understanding with Persistent Event Memory (NeurIPS 2025 · similarity 0.87)
- EventGPT: Event Stream Understanding with Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
- PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling (NeurIPS 2025 · similarity 0.87)
- MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval (CVPR 2025 · similarity 0.86)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.86)
- EventLens: Event-Structure Reinforcement Learning for Video Understanding (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.