Research map / Video / Video-language understanding
ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the BestShot task and benchmark for language-query highlight frame retrieval in human-centric videos, plus two synthetic training datasets (ShotGPT4o, Image-SMPLText), and fine-tunes InternVL into ShotVL.
Problem
Locating highlight frames in human-centric videos via detailed language queries (content, fine-grained action stage, human pose) requires frame-level precision that existing moment retrieval, TAL, and CLIP/InternVL-style models lack, partly due to coarse annotations and limited training data.
Approach
Combine a human-annotated frame-level benchmark with two large synthetic datasets (GPT-4o-generated frame Q&As and SMPL-based pose descriptions adapted from PoseScript on real video datasets) and fine-tune InternVL on them, mixed with general image-caption data to retain general capability.
Key results
- ShotVL reaches 53.4 top-1 on the full BestShot query vs 32.6 for InternVL (about 52% relative improvement reported).
- On THUMOS14 zero-shot TAL, ShotVL reaches 13.17 avg mAP vs 8.37 for InternVL.
- ShotVL keeps general CLIP classification average at 75.90 vs 77.70 for InternVL, with notable drops on ImageNet variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune InternVL as ShotVL on the proposed training data | InternVL 14B · BestShot Full | Top@1: 32.6 -> 53.4 | Helped |
| fine-tune InternVL as ShotVL on the proposed training data | InternVL 14B · THUMOS14 | avg mAP: 8.37 -> 13.17 | Helped |
| fine-tune InternVL on 2,000 human-written pose descriptions (ablation) | InternVL · BestShot Full | Top@1: 32.6 -> 40.5 | Helped |
| train with Image-SMPLText pose descriptions (ablation) | InternVL · BestShot Pose | Top@1: 20.6 -> 33.2 | Helped |
| add human-written pose descriptions to Image-SMPLText (ablation) | ShotVL-PoseS · BestShot Full | Top@1: 40.8 -> 48.6 | Helped |
| add GPT-generated pose descriptions to human-written and Image-SMPLText pose data (ablation) | ShotVL-PoseSH · BestShot Full | Top@1: 48.6 -> 49.4 | No clear effect |
| add COCO general image-caption data (ablation) | ShotVL-G trained with LAION · BestShot Full | Top@1: 48.9 -> 51.5 | Helped |
| add ShotGPT4o data to pose and COCO training (ablation) | ShotVL-G (COCO) · BestShot Full | Top@1: 51.5 -> 53.4 | Helped |
| add ShotGPT4o data to LITA training (ablation) | LITA · BestShot Full | Avg. IoU score: 6.64 -> 10.60 | Helped |
| add the paper's data to LITA training (ablation) | LITA · Charades | score at IoU>0.7: 4.005 -> 6.226 | Helped |
| replace InternVL with ShotVL while training LITA on frame-retrieval data (ablation) | InternVL-LITA-FR · BestShot Pose | Avg. IoU score: 6.87 -> 9.92 | Helped |
| fine-tune ShotVL using pose and general training data (ablation) | InternVL · CLIP classification benchmark avg(C) | Top@1: 77.70 -> 75.90 | Hurt |
Surprising findings
- GPT-4o pose descriptions were only about 70% accurate (confusing left and right), so training on them alone performed poorly; the authors thus relied on SMPL-based descriptions.
- Adding Image-SMPLText markedly reduced general retrieval ability, requiring mixing in general image-text data.
Limitations
- Benchmark concentrates on limited categories, mostly sports.
- ShotVL-LITA video extension declined on BestShot, especially for actions, attributed to limited video training data and token compression.
- GPT-4o-generated annotations are noisy; pose descriptions suffer from left-right confusion and occlusion errors.
Experimental setup
Datasets: ShotGPT4o Image-SMPLText LAION-400M Kinetics-700 COCO captions PoseScript THUMOS14 FineAction Flickr30K AVA-Kinetics
Benchmarks: BestShot Benchmark THUMOS14 AVA-Kinetics Flickr30K CLIP classification benchmark (24 tasks) Charades-STA
Metrics: top-1 accuracy mAP accuracy top1/top5 IoU-based recall
Baselines: CLIP ViT-B/32 CLIP ViT-L/14 LongCLIP InternVL InternVideo VTG-LLM LITA
Builds on: InternVL PoseScript T3AL LITA LongCLIP
Closest papers by meaning
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.87)
- Anchor-Aware Similarity Cohesion in Target Frames Enables Predicting Temporal Moment Boundaries in 2D (CVPR 2025 · similarity 0.86)
- MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval (NeurIPS 2025 · similarity 0.86)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.86)
- Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation (NeurIPS 2026 · similarity 0.86)
- TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.