Research map / Video / Action recognition
Open-Vocabulary Video Relation Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces open-vocabulary video relation extraction (OVRE), a dataset (Moments-OVRE, ~187K videos) of action-centric relation triplets, and a CLIP-ViT + attentional pooler + GPT-2 generative baseline evaluated with caption-style metrics.
Problem
Existing video relation detection relies on fixed object and predicate vocabularies and object-centric relations, so it fails to describe the actors and relations that constitute an action in open-vocabulary language.
Approach
Reframes action understanding as generating open-vocabulary, action-centric relation triplets from raw video with a generative vision-to-text model, and matches generated triplet sets to ground truth via SimCSE similarity and Hungarian matching before scoring with BLEU, CIDEr, and METEOR.
Key results
- The proposed CLIP-ViT + attentional pooler + GPT-2 model reaches CIDEr 174.47 and METEOR 25.07 on Moments-OVRE, versus GIT (155.38 / 23.06) and ClipCap (125.45 / 19.25).
- Fine-tuning CLIP-ViT improves CIDEr from 165.67 to 174.47; fine-tuning GPT-2 raises CIDEr from 131.85 to 165.67.
- Patch features (CIDEr 165.67) outperform frame (115.85) and region features (77.38) when the visual encoder is frozen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed CLIP-ViT and GPT-2 relation-generation method instead of GIT | Moments-OVRE generation · Moments-OVRE test | METEOR: 23.06 -> 25.07; +2.01 | Helped |
| use the proposed CLIP-ViT and GPT-2 relation-generation method instead of ClipCap | Moments-OVRE generation · Moments-OVRE test | METEOR: 19.25 -> 25.07; +6.22 | Helped |
| fine-tune GPT-2 rather than freeze both GPT-2 and CLIP (ablation) | OVRE model · Moments-OVRE test | CIDEr: 131.85 -> 165.67 | Helped |
| fine-tune CLIP-ViT in addition to GPT-2 (ablation) | OVRE model · Moments-OVRE test | CIDEr: 165.67 -> 174.47; 8.8 | Helped |
| use patch features instead of frame features (ablation) | OVRE visual input · Moments-OVRE test | CIDEr: 115.85 -> 165.67 | Helped |
| use patch features instead of region features (ablation) | OVRE visual input · Moments-OVRE test | CIDEr: 77.38 -> 165.67 | Helped |
| increase the query count from 2 to 8 (ablation) | OVRE attentional pooler · Moments-OVRE test | CIDEr and METEOR | Helped |
| increase query count beyond 8 by doubling it (ablation) | OVRE attentional pooler · Moments-OVRE test | CIDEr and METEOR | No clear effect |
| sort relation triplets using Triplet Linearization rather than concatenate them in arbitrary order (ablation) | OVRE text decoder · Moments-OVRE | generation performance | No clear effect |
Surprising findings
- Region features from an object detector with tracking performed worse than frame-level features, which the authors attribute to limited detector generalization to diverse categories.
- Sorting triplets by triplet linearization did not provide significant benefits, since relations are treated as unordered.
Limitations
- No bounding box annotations in the current Moments-OVRE release due to annotation cost.
- Evaluation relies on n-gram and embedding-similarity metrics rather than triplet-level precision/recall, and no human evaluation is reported.
- Videos average about 3 seconds long, which limits temporal context.
Experimental setup
Datasets: Moments-OVRE Multi-Moments in Time (M-MiT)
Benchmarks: Moments-OVRE
Metrics: BLEU@1 BLEU@2 BLEU@3 CIDEr METEOR
Baselines: ClipCap GIT
Builds on: ClipCap VideoCoCa CLIP GIT
Closest papers by meaning
- Does Video-Text Pretraining Help Open-Vocabulary Online Action Detection? (NeurIPS 2024 · similarity 0.88)
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.86)
- Storyboard-guided Alignment for Fine-grained Video Action Recognition (NeurIPS 2025 · similarity 0.86)
- Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition (NeurIPS 2025 · similarity 0.86)
- From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding (CVPR 2024 · similarity 0.86)
- Multi-Modal Prompting for Open-Vocabulary Video Visual Relationship Detection (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.