Research map / Video / Video-language understanding
Multi-Modal Prompting for Open-Vocabulary Video Visual Relationship Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts CLIP to open-vocabulary video visual relationship detection via spatio-temporal visual prompting in the image encoder and vision-guided continuous/conditional language prompts, evaluated on VidVRD and VidOR.
Problem
Open-vocabulary video visual relationship detection requires recognizing unseen subject-predicate-object triplets in videos, but image-text models like CLIP lack object-level spatio-temporal context and do not transfer well to discovering novel relationship predicates.
Approach
Prompt CLIP on both sides: spatio-temporal visual prompting (spatial and temporal Transformer blocks over masked subject, object, union and background frames) fed into the frozen image encoder, plus vision-guided language prompts that mix learnable continuous tokens with tokens generated from visual features by a small prompting network.
Key results
- On VidVRD novel split with detected trajectories, the method reaches 15.99 mAP vs 6.10 for RePro, nearly 10 points gain.
- On VidOR with ground-truth trajectories, the All split gains 6.8% R@50 and 8.04% R@100 over the best baseline.
- Ablation on VidVRD shows removing both visual and language prompting lowers average mAP from 25.57 to 14.50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply multi-modal visual and language prompting instead of neither (ablation) | CLIP for Open-VidVRD · VidVRD Novel, DET trajectory | mAP: 5.03 -> 15.99 | Helped |
| add visual prompting to language prompting (ablation) | CLIP for Open-VidVRD · VidVRD Novel, DET trajectory | mAP: 12.33 -> 15.99 | Helped |
| add language prompting to visual prompting (ablation) | CLIP for Open-VidVRD · VidVRD Novel, DET trajectory | mAP: 11.04 -> 15.99 | Helped |
| add spatial and temporal modeling (ablation) | CLIP visual prompting · VidVRD Novel, DET trajectory | mAP: 12.33 -> 15.99 | Helped |
| add temporal modeling to spatial modeling (ablation) | CLIP visual prompting · VidVRD Novel, DET trajectory | mAP: 13.92 -> 15.99 | Helped |
| add spatial modeling to temporal modeling (ablation) | CLIP visual prompting · VidVRD Novel, DET trajectory | mAP: 9.10 -> 15.99 | Helped |
| combine learnable continuous and conditional language prompts (ablation) | CLIP language prompting · VidVRD Novel, DET trajectory | mAP: 14.21 -> 15.99 | Helped |
| replace manually designed templates with vision-guided language prompting (ablation) | CLIP language prompting · VidVRD Novel, DET trajectory | mAP: 11.04 -> 15.99 | Helped |
| use MMP instead of RePro | Open-VidVRD · VidVRD Novel, DET trajectory | mAP: 6.10 -> 15.99 | Helped |
| use MMP instead of RePro | Open-VidVRD · VidVRD Novel, GT trajectory | mAP: 12.74 -> 21.14 | Helped |
| use MMP instead of RePro | Open-VidVRD · VidVRD All, DET trajectory | mAP: 21.33 -> 27.06 | Helped |
| use MMP instead of RePro | Open-VidVRD · VidVRD All, GT trajectory | mAP: 34.90 -> 38.08 | Helped |
Surprising findings
- Temporal modeling alone without spatial modeling causes a large performance drop (average mAP 19.07 vs 25.57 full), suggesting spatial context matters more than temporal dynamics alone.
Limitations
- Depends on a pre-trained trajectory detector; not end-to-end.
- Performance on the novel split with detected trajectories is limited (mAP about 16 on VidVRD).
Experimental setup
Datasets: VidVRD VidOR
Benchmarks: VidVRD VidOR
Metrics: mAP Recall@50 Recall@100
Baselines: RePro VidVRD-II ALPro CLIP MaPLe
Builds on: CLIP MaPLe RePro VidVRD-II
Closest papers by meaning
- Towards Flexible Visual Relationship Segmentation (NeurIPS 2024 · similarity 0.87)
- Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation (AAAI 2025 · similarity 0.87)
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.87)
- Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation (NeurIPS 2024 · similarity 0.87)
- Remodeling Semantic Relationships in Vision-Language Fine-Tuning (AAAI 2026 · similarity 0.86)
- STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.