Research map / Video / Video-language understanding
End-To-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add end-to-end concept-word detection and semantic attention to the base model | Base-SAN · LSMDC16 movie description | BLEU-4: 0.005 -> 0.008 | Helped |
| Add end-to-end concept-word detection and semantic attention to the base model | Base-SAN · LSMDC16 fill-in-the-blank | accuracy: 34.5 -> 41.9 | Helped |
| Add end-to-end concept-word detection and semantic attention to the base model | Base-SAN · LSMDC16 multiple-choice | accuracy: 60.1 -> 63.8 | Helped |
| Add end-to-end concept-word detection and semantic attention to the base model | Base-SAN · LSMDC16 movie retrieval | Recall@1: 4.3 -> 4.5 | Helped |
| Replace detected concept words with randomly sampled words (ablation) | CT-SAN · LSMDC16 fill-in-the-blank | accuracy: 41.9 -> 17.0 | Hurt |
| Replace spatial-attention concept detection with a single LSTM on pooled video features (ablation) | CT-SAN · LSMDC16 fill-in-the-blank | accuracy: 41.9 -> 37.4 | Hurt |
| Train the concept detector separately instead of jointly end-to-end (ablation) | CT-SAN · LSMDC16 fill-in-the-blank | accuracy: 41.9 -> 35.4 | Hurt |
| Use nearest-neighbor video concept words instead of detected words (ablation) | CT-SAN · LSMDC16 fill-in-the-blank | accuracy: 41.9 -> 12.1 | Hurt |
| Vary the number of detected concept words from 10 to 5 or 20 (ablation) | CT-SAN · LSMDC16 movie description | BLEU-4: 0.007, 0.008, 0.008 | No clear effect |
| Remove output semantic attention (ablation) | CT-SAN · LSMDC16 movie description | BLEU-4: 0.008 -> 0.006 | Hurt |
| Remove input semantic attention (ablation) | CT-SAN · LSMDC16 fill-in-the-blank | accuracy: 41.9 -> 39.1 | Hurt |
| Ensemble six independently trained models | CT-SAN multiple-choice · LSMDC16 multiple-choice | accuracy: 63.8 -> 67.0 | Helped |
Closest papers by meaning
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- On Scaling Up a Multilingual Vision and Language Model (CVPR 2024 · similarity 0.86)
- Set Prediction Guided by Semantic Concepts for Diverse Video Captioning (AAAI 2024 · similarity 0.86)
- OmniViD: A Generative Framework for Universal Video Understanding (CVPR 2024 · similarity 0.86)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.86)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.