Research map / Video / Video-language understanding
Video Question Answering with Iterative Video-Text Co-Tokenization
AJ Piergiovanni, Kairo Morton, Weicheng Kuo, Michael S. Ryoo, Anelia Angelova
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add iterative video-text co-tokenization (ablation) | 2-stream + Tok + MS · MSRVTT-QA (open vocab, trained from scratch) | accuracy: 26.2 -> 27.6 | Improved |
| add iterative video-text co-tokenization (ablation) | 2-stream + Tok + MS · TGIF Frame-QA (open vocab, from scratch) | accuracy: 26.2 -> 27.3 | Improved |
| add multi-scale features (ablation) | 2-stream + Tok · MSRVTT-QA (open vocab, trained from scratch) | accuracy: 25.5 -> 26.2 | Improved |
| add learned tokenization (ablation) | 2-stream + Transformer · MSRVTT-QA (open vocab, trained from scratch) | accuracy: 25.3 -> 25.5 | Improved |
| add transformer fusion (ablation) | 2-stream · MSRVTT-QA (open vocab, trained from scratch) | accuracy: 24.7 -> 25.3 | Improved |
| use two streams instead of one stream (ablation) | single-stream · MSRVTT-QA (open vocab, trained from scratch) | accuracy: 24.8 -> 24.7 | No clear effect |
| use multiple frames instead of a single frame (ablation) | video encoder · TGIF Frame-QA (open vocab, from scratch) | accuracy: 24.4 -> 21.5 | Hurt |
| use masked 4k vocabulary instead of open 32k vocabulary (ablation) | Ours · MSRVTT-QA | accuracy: 33.7 -> 45.7 | Improved |
| use masked vocabulary instead of FC vocabulary (ablation) | Ours · MSRVTT-QA | accuracy: 42.9 -> 45.7 | Improved |
| use generative instead of contrastive pretraining on HowTo100M (ablation) | single-stream X3D-M · MSRVTT-QA (single-stream X3D-M, pretraining comparison) | accuracy: 26.2 -> 29.4 | Improved |
| pretrain on HowTo100M instead of YouTube8M (generative) (ablation) | single-stream X3D-M · MSRVTT-QA (single-stream X3D-M, pretraining comparison) | accuracy: 27.8 -> 29.4 | Improved |
| add pretraining versus random init (ablation) | single-stream X3D-M · MSRVTT-QA (single-stream X3D-M, pretraining comparison) | accuracy: 8.3 -> 29.4 | Improved |
Closest papers by meaning
- Granularity-Adaptive Spatial Evidence Tokenization for Video Question Answering (AAAI 2025 · similarity 0.88)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.88)
- Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models (CVPR 2025 · similarity 0.88)
- Streaming Video Question-Answering with In-context Video KV-Cache Retrieval (ICLR 2025 · similarity 0.87)
- Video-Context Aligned Transformer for Video Question Answering (AAAI 2024 · similarity 0.87)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
