Research map / Video / Video-language understanding
Efficient End-to-End Video Question Answering with Pyramidal Multimodal Transformer
Min Peng, Chongyang Wang, Yu Shi, Xiang-Dong Zhou
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the spatial-temporal decomposition to the pyramid (ablation) | PMT · TGIF-QA | Action accuracy: 85.4 -> 87.6 | Improved |
| Use contextual matching instead of up-sampling in the top-down pathway (ablation) | PMT · TGIF-QA | Action accuracy: 86.7 -> 87.6 | Improved |
| Use contextual matching instead of attention-based fusion in the top-down pathway (ablation) | PMT · TGIF-QA | Action accuracy: 86.5 -> 87.6 | Improved |
| Add the multistep loss constraint (ablation) | PMT · TGIF-QA | Action accuracy: 87.5 -> 87.6 | Improved |
| Replace default encoders with pre-trained TimeSformer and BERT (ablation) | PMT · MSRVTT-QA | accuracy: +0.3 | Improved |
| Use 16 input frames rather than other tested frame counts (ablation) | PMT · MSVD-QA and MSRVTT-QA | accuracy | Mixed |
| Set pyramid depth to L=3 instead of other tested depths (ablation) | PMT · MSVD-QA | accuracy | No clear effect |
Closest papers by meaning
- Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering (CVPR 2019 · similarity 0.89)
- Contrastive Video Question Answering via Video Graph Transformer (TPAMI 2023 · similarity 0.89)
- Just Ask: Learning To Answer Questions From Millions of Narrated Videos (ICCV 2021 · similarity 0.89)
- Video-Context Aligned Transformer for Video Question Answering (AAAI 2024 · similarity 0.88)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.87)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
