Research map / Video / Video-language understanding

Localizing Events in Videos with Multimodal Queries

CVPR 2025 · Video › Video-language understanding · approach: Video LLMs with fine-grained understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ICQ, a benchmark (with ICQ-Highlight built on QVHighlights) for localizing video events with multimodal queries (reference image + refinement text), and adapts 12 NLQ-based localization models via training-free MQ adaptation and a surrogate fine-tuning strategy (SUIT).

Problem

Video event localization has been studied almost exclusively with natural language queries, leaving multimodal queries that combine images and text (for unfamiliar or hard-to-verbalize concepts) unexplored, and existing NLQ-based models do not directly accept such queries.

Approach

Formulate multimodal queries as a reference image plus refinement text, and adapt NLQ-based video localization models to them either by converting MQs into text queries (MQ-Captioning, MQ-Summarization, with SUIT fine-tuning on pseudo-MQs from image-caption data) or by encoding the reference image with the CLIP visual encoder (VQ-Enc).

Key results

• MQ-Sum with SUIT improves over training-free MQ-Sum by about 4.3%-9.7% and gives more stable performance across backbones.
• Among training-free methods, MQ-Cap performs best, outperforming other adaptation methods by an average margin of about 3.6%.
• Performance on ICQ-Highlight correlates highly with NLQ-based QVHighlights performance, and a controlled experiment with retrieved real images shows generation artifacts do not largely skew conclusions.

Measured effects reported in the paper

Surprising findings

• Scribble reference images perform close to other styles across models, with EaTR even slightly exceeding its cartoon/realistic-free NLQ-like counterpart on some metrics.
• LLM-based models (SeViLA, TimeChat, VTimeLLM) lag far behind specialized models on the MQ benchmark.

Limitations

• Reference images are synthetic (generated by text-to-image models) and contain artifacts, though a controlled check suggests limited impact.
• Large performance gap remains between MQ setting and the original NLQ benchmark, indicating query semantics are distorted across modalities.
• Adaptation relies on existing NLQ-based models rather than architectures designed for multimodal queries.

Experimental setup

Datasets: QVHighlights Flickr30K COCO

Benchmarks: ICQ-Highlight

Metrics: Recall@1 at IoU 0.5 Recall@1 at IoU 0.7 mean Average Precision

Baselines: Moment-DETR QD-DETR EaTR CG-DETR TR-DETR UMT UniVTG UVCOM SeViLA TimeChat VTimeLLM

Builds on: QVHighlights Moment-DETR LoRA CLIP

Closest papers by meaning

• Transferable Video Moment Localization by Moment-Guided Query Prompting (AAAI 2024 · similarity 0.88)
• Boundary-Aware Temporal Dynamic Pseudo-Supervision Pairs Generation for Zero-Shot Natural Language Video Localization (AAAI 2025 · similarity 0.87)
• Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
• RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations (CVPR 2025 · similarity 0.87)
• Unsupervised Domain Adaptative Temporal Sentence Localization with Mutual Information Maximization (AAAI 2024 · similarity 0.86)
• Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.86)