Research map / Vision-language and multimodal / CLIP and zero-shot
Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Moment-GPT performs tuning-free zero-shot video moment retrieval by debiasing queries with LLaMA-3, generating candidate spans from MiniGPT-v2 frame captions, and scoring spans with Video-ChatGPT captions and LLaMA-3 similarity.
Problem
Video moment retrieval with frozen MLLMs suffers from language bias in human-annotated queries (rare words, spelling and grammar errors) and existing zero-shot pipelines produce inaccurate spans, while fine-tuned MLLM methods need costly annotated data.
Approach
Combine LLM-based query debiasing (correction and multiple rewrites), caption-based frame similarity with an adaptive histogram-threshold span generator, and span-level video captions from a video MLLM scored against debiased queries, all without any weight updates.
Key results
- On QVHighlights test, Moment-GPT reaches R1@0.5 58.3 and mAP@avg 35.0, improving over the prior zero-shot SOTA by at least 3.4 points per metric.
- On Charades-STA, Moment-GPT achieves R1@0.5 38.4 and mIoU 36.5, outperforming the prior zero-shot SOTA by 4.8 points in R1@0.3.
- On ActivityNet-Captions, Moment-GPT reaches R1@0.5 31.1 and outperforms the prior zero-shot SOTA by 2.5 points in R1@0.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the strongest reported zero-shot baseline with Moment-GPT | QVHighlights test · QVHighlights test | R1@0.5: 52.4 -> 58.3 | Helped |
| Replace the strongest reported zero-shot baseline with Moment-GPT | QVHighlights val · QVHighlights val | R1@0.5: 53.1 -> 58.9 | Helped |
| Replace the strongest reported zero-shot baseline with Moment-GPT | Charades-STA · Charades-STA | R1@0.5: 36.0 -> 38.4 | Helped |
| Replace the strongest reported zero-shot baseline with Moment-GPT | ActivityNet-Captions · ActivityNet-Captions | R1@0.5: 27.4 -> 31.1 | Helped |
| Use LLaMA-3 instead of LLaMA-2 or Mistral-7B for query debiasing (ablation) | Moment-GPT · Charades-STA | R1@0.5: 37.7 -> 38.4; 37.2 -> 38.4 | Helped |
| Use MiniGPT-v2 for image captioning instead of LLaVA or MiniGPT-4 (ablation) | Moment-GPT · Charades-STA | R1@0.5: 37.1 -> 38.4; 37.8 -> 38.4 | Helped |
| Use LLaMA-3 rather than BLIP-2, BLIP-2-T, BERT, or SimCSE for frame scoring (ablation) | Moment-GPT · Charades-STA | R1@0.5: 34.8 -> 38.4; 36.9 -> 38.4; 37.3 -> 38.4; 38.0 -> 38.4 | Helped |
| Use the adaptive span generator instead of sliding windows or shot detection (ablation) | Moment-GPT · Charades-STA | R1@0.5: 27.4 -> 38.4; 32.1 -> 38.4 | Helped |
| Use Video-ChatGPT video captions rather than no captions or MiniGPT-v2 image captions (ablation) | Moment-GPT · Charades-STA | R1@0.5: 30.2 -> 38.4; 32.4 -> 38.4 | Helped |
| Use 3 debiased queries instead of only the raw query (ablation) | Moment-GPT · Charades-STA | R1@0.5: 36.9 -> 38.4 | Helped |
| Generate 1 rather than 3 debiased queries (ablation) | Moment-GPT · Charades-STA | R1@0.5: 38.4 -> 36.3 | Hurt |
| Generate 4 or 5 rather than 3 debiased queries (ablation) | Moment-GPT · Charades-STA | R1@0.5: 38.4 -> 38.1; 38.4 -> 37.4 | Hurt |
Surprising findings
- Replacing BLIP-2 frame similarity with MiniGPT-v2 captions followed by a text LLM similarity raised Charades-STA R1@0.5 from 34.8 to 38.4 with the same stack.
- Video-ChatGPT span captions outperformed image MLLM (MiniGPT-v2) span captions, which gave poor results.
Limitations
- Relies on converting video to text captions and text similarity, losing information and possibly introducing encoding bias
- Uses three MLLMs, raising inference time and GPU memory compared with single-model baselines
- Performance constrained by current MLLM VMR ability; no open-source unified video-text embedding MLLM
Experimental setup
Datasets: QVHighlights Charades-STA ActivityNet-Captions
Benchmarks: QVHighlights Charades-STA ActivityNet-Captions
Metrics: R1@0.5 R1@0.7 mAP@0.5 mAP@avg mIoU HIT@1
Baselines: Moment-DETR UMT MH-DETR VTimeLLM LLaViLo VideoLLaMA VideoChatGPT UniVTG Wattasseril et al. TimeChat GroundingGPT CNM
Builds on: Video-ChatGPT MiniGPT-v2 LLaMA-3 Luo et al. zero-shot VMR
Closest papers by meaning
- Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language (AAAI 2024 · similarity 0.91)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.90)
- VERIFIED: A Video Corpus Moment Retrieval Benchmark for Fine-Grained Video Understanding (NeurIPS 2024 · similarity 0.89)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.89)
- When One Moment Isn't Enough: Multi-Moment Retrieval with Cross-Moment Interactions (NeurIPS 2025 · similarity 0.88)
- Boundary-Aware Temporal Dynamic Pseudo-Supervision Pairs Generation for Zero-Shot Natural Language Video Localization (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.