Research map / Video / Video-language understanding
Open Vocabulary Multi-Label Video Classification
Rohit Gupta, Mamshad Nayeem Rizve, Jayakrishnan Unnikrishnan, Ashish Tawari, Son Tran, Mubarak Shah, Benjamin Yao, Trishul A Chilimbi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed LLM prompts with learnable LLM prompts and prompt transformer (ablation) | CLIP label encoder · ActivityNet (open vocab) | AUPR: 48.1 -> 50.2 | Improved |
| add prompt transformer to the fixed-prompt LLM label encoder (ablation) | CLIP label encoder · ActivityNet (open vocab) | AUPR: 41.3 -> 48.1 | Improved |
| add an LLM to the CLIP text encoder label encoder (ablation) | label encoder · ActivityNet (open vocab) | AUPR: 34.5 -> 50.2 | Improved |
| keep the CLIP text encoder in the label encoder (ablation) | label encoder · TAO (open vocab) | AUPR: 25.2 -> 65.5 | Improved |
| add parallel temporal modeling branch | CLIP + Learnable LLM Prompts · TAO (open vocab) | AUPR: 51.4 -> 63.8 | Improved |
| add temporal modeling | CLIP + Learnable LLM Prompts · ActivityNet (open vocab) | AUPR: 47.1 -> 47.1 | No clear effect |
| add stochastic weight regularization on temporal-branch spatial layers (ablation) | temporal modeling branch · ActivityNet (open vocab) | Peak F1: 32.5 -> 53.8 | Improved |
| unfreeze the CLIP vision backbone (no regularization) (ablation) | temporal modeling branch · TAO (open vocab) | Peak F1: 62.1 -> 42.7 | Hurt |
| use more temporal blocks up to 4 (ablation) | temporal modeling branch · ActivityNet (open vocab) | AUPR: 47.7 (1 block) -> 50.2 (4) -> 48.1 (5) | Mixed |
| use parallel instead of serial temporal blocks (ablation) | temporal modeling branch · ActivityNet (open vocab) | Peak F1: 41.5 -> 52.6 | Improved |
| add synthetic labels from a multimodal LLM to training data | full method · ActivityNet (open vocab) | AUPR: 47.1 -> 50.2 | Improved |
| add synthetic labels | full method · TAO (open vocab) | Peak F1: 59.6 -> 56.6 | Hurt |
Closest papers by meaning
- Towards Open Vocabulary Learning: A Survey (TPAMI 2024 · similarity 0.90)
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.89)
- OVMR: Open-Vocabulary Recognition with Multi-Modal References (CVPR 2024 · similarity 0.87)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.87)
- Does VLM Classification Benefit from LLM Description Semantics? (AAAI 2025 · similarity 0.87)
- Recover and Match: Open-Vocabulary Multi-Label Recognition through Knowledge-Constrained Optimal Transport (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
