Research map / Video / Video-language understanding
Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videos
Dhruv Verma, Debaditya Roy, Basura Fernando
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CoFormer with ClipSitu XTF using CLIP embeddings | imSitu test, top-1 predicted verb · imSitu | value: 35.98 -> 47.21 | Improved |
| use ClipSitu Verb MLP instead of linear probe on CLIP | ViT-L14@336px verb prediction · imSitu | verb top-1: 53.4 -> 57.9 | Improved |
| use ClipSitu Verb MLP instead of fine-tuning CLIP with wise-ft | ViT-L14@336px verb prediction · imSitu | verb top-1: 52.2 -> 57.9 | Improved |
| use 2 hidden layers instead of 1 in Verb MLP (ablation) | ClipSitu Verb MLP, ViT-L14@336px · imSitu | verb top-1: 57.9 -> 56.2 | Hurt |
| use 3 hidden layers instead of 1 in Verb MLP (ablation) | ClipSitu Verb MLP, ViT-L14@336px · imSitu | verb top-1: 57.9 -> 54.3 | Hurt |
| use minimum annotator cross-entropy loss instead of per-annotator cross-entropy (ablation) | ClipSitu MLP · imSitu | value: 76.57 -> 76.91 | Improved |
| use CLIP embeddings instead of ALIGN (ablation) | ClipSitu XTF · imSitu dev | value (top-1 verb): 31.17 -> 47.23 | Improved |
| use CLIP text embeddings instead of Glove for verbs and roles (ablation) | ClipSitu XTF · imSitu dev | value (top-1 verb): 42.40 -> 47.23 | Improved |
| increase patch tokens via larger encoder ViT-L14@336px instead of ViT-B32 (ablation) | ClipSitu XTF · imSitu ground-truth verb | value: 75.2 -> 78.5 | Improved |
| add cross-attention scores to verb-role embeddings for noun localization (ablation) | ClipSitu XTF · imSitu ground-truth verb | grounded value: 27.81 -> 55.36 | Improved |
| use cross-attention scores from first layer instead of last layer (ablation) | ClipSitu XTF · imSitu top-1 verb | grounded value: 33.30 -> 39.30 | Improved |
| use XTF encoder instead of MLP with decoder on video (ablation) | ClipSitu+TxD, X-CLIP ViT-L14 · VidSitu | CIDEr: 61.93 -> 49.65 | Hurt |
Closest papers by meaning
- Situation Recognition: Visual Semantic Role Labeling for Image Understanding (CVPR 2016 · similarity 0.91)
- Grounded Video Situation Recognition (NeurIPS 2022 · similarity 0.89)
- Visual Semantic Role Labeling for Video Understanding (CVPR 2021 · similarity 0.88)
- OmniViD: A Generative Framework for Universal Video Understanding (CVPR 2024 · similarity 0.84)
- ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.84)
- Transferable Video Moment Localization by Moment-Guided Query Prompting (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
