Research map / Video / Video-language understanding
Training-free Online Video Step Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BaGLM, a training-free online method for video step grounding that prompts an off-the-shelf LMM per 2-second segment and refines its scores with Bayesian filtering using LLM-derived step dependencies and LMM-estimated progress.
Problem
Video step grounding (detecting which procedural steps occur in a video) is usually solved with training-based, offline methods that need labeled data and full-video access, which limits generalization and online use.
Approach
Use zero-shot LMMs to score steps from the current segment only, then inject past-frame knowledge via a Bayesian filter whose transition matrix comes from LLM-estimated prerequisite dependencies and LMM-estimated step progress (readiness and validity).
Key results
- BaGLM reaches 57.4 R@1 on HT-Step, outperforming offline NaSVA (53.1) and the online NaSVA variant (46.1).
- BaGLM achieves 59.8 Avg. R@1 on CrossTask, versus 47.9 for offline MPTVA.
- BaGLM obtains 43.3 R@1 on Ego4D Goal-Step, versus 29.1 for NaSVA and 24.3 for VSLNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Bayesian filtering (BaGLM) on top of zero-shot InternVL2.5-8B | InternVL2.5-8B · HT-Step | R@1: +2.4 (55.0 to 57.4 range; 52.0 off-the-shelf to 57.4) | Helped |
| Add Bayesian filtering (BaGLM) on top of zero-shot InternVL2.5-8B | InternVL2.5-8B · CrossTask | Avg. R@1: 54.3 to 59.8 | Helped |
| Add Bayesian filtering (BaGLM) on top of zero-shot InternVL2.5-8B | InternVL2.5-8B · Ego4D Goal-Step | R@1: 45.2 to 43.3 (-1.9) | No clear effect |
| Add readiness score to static transition matrix (ablation) | BaGLM transition model · HT-Step | R@1: +1.1 (55.9 to 57.0) | Helped |
| Add validity score to static transition matrix (ablation) | BaGLM transition model · HT-Step | R@1: +0.5 (55.9 to 56.4) | Helped |
| Add readiness score to static transition matrix (ablation) | BaGLM transition model · Ego4D Goal-Step | R@1: 42.1 to 42.0 | No clear effect |
| Add both readiness and validity to transition model (ablation) | BaGLM transition model · CrossTask | Avg. R@1: +1.8 (58.0 to 59.8) | Helped |
| Use oracle step progress from annotations (ablation) | BaGLM · CrossTask | Avg. R@1: 59.8 to 38.6 (oracle progress only in Table 4, dependency not oracle) | Hurt |
| Use oracle dependency matrix and oracle progress together (ablation) | BaGLM · Ego4D Goal-Step | R@1: 43.3 to 82.2 | Helped |
| Replace LLaMA-3.3-70B with GPT-4.1-mini for dependency matrix (ablation) | BaGLM · Ego4D Goal-Step | R@1: 43.3 to 40.6 | Hurt |
| Replace LLaMA-3.3-70B with GPT-4.1-mini for dependency matrix (ablation) | BaGLM · CrossTask | Avg. R@1: 59.8 to 60.9 | Helped |
| Use multi-choice VSG prompt instead of binary per-step prompt (ablation) | InternVL2.5-8B · Ego4D Goal-Step | R@1: +3.4 (41.8 to 45.2) | Helped |
Surprising findings
- Off-the-shelf InternVL2.5-8B, seeing only the current 2-second segment, already matches or beats several training-based offline methods on CrossTask and Ego4D Goal-Step.
- Using a uniform prior over steps in the update step outperformed context-dependent next-step priors.
Limitations
- Relies entirely on a pre-trained LLM for the step dependency matrix, so quality depends on that LLM.
- Step progress estimation is done by querying an LMM and does not use priors on step durations.
- Gains are limited on Ego4D Goal-Step due to long videos and coarse, generic step descriptions.
Experimental setup
Datasets: HT-Step CrossTask Ego4D Goal-Step COIN
Benchmarks: HT-Step CrossTask Ego4D Goal-Step COIN
Metrics: Recall@1 Avg. Recall@1 mAP@IoU
Baselines: Zhukov et al. HT100M VideoCLIP MCN DWSA MIL-NCE VT-TWINS UniVL VINA TAN* NaSVA MPTVA
Builds on: Bayesian filtering NaSVA VINA MPTVA
Closest papers by meaning
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.87)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.87)
- Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (NeurIPS 2025 · similarity 0.86)
- TRACE: Temporal Grounding Video LLM via Causal Event Modeling (ICLR 2025 · similarity 0.86)
- Uncertainty-quantified Rollout Policy Adaptation for Unlabelled Cross-domain Video Temporal Grounding (NeurIPS 2025 · similarity 0.86)
- SnAG: Scalable and Accurate Video Grounding (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.