Research map / Vision-language and multimodal / Visual question answering
Learning Visual Question Answering by Bootstrapping Hard Attention
Mateusz Malinowski, Carl Doersch, Adam Santoro, Peter Battaglia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select only 16% of cells by L2 norm instead of all cells (sum pooling) (ablation) | sum pooling · VQA-CP v2 | overall accuracy: 27.96 -> 26.99 | No clear effect |
| select 48% of cells with hard attention instead of all cells (pairwise aggregation) (ablation) | non-local pairwise · VQA-CP v2 | overall accuracy: 28.06 -> 28.23 | Improved |
| use soft attention instead of sum pooling | sum pooling · VQA-CP v2 | overall accuracy: 27.96 -> 26.60 | No clear effect |
| add adaptive threshold selection (AdaHAN) to pairwise aggregation instead of fixed 48% cells | HAN+pairwise · VQA-CP v2 | overall accuracy: 28.23 -> 28.65 | Improved |
| add adaptive selection (AdaHAN) to sum pooling vs HAN 32% | HAN+sum · VQA-CP v2 | overall accuracy: 27.43 -> 27.40 | No clear effect |
| add the two alignment layers before attention (HAN+sum 25%) (ablation) | HAN+sum · VQA-CP v2 | overall accuracy: 26.38 -> 26.99 | Improved |
| add the two alignment layers before attention (HAN+sum 75%) (ablation) | HAN+sum · VQA-CP v2 | overall accuracy: 26.82 -> 27.94 | Improved |
| replace soft attention with HAN when training CNN from scratch | simple-SAN · VQA-CP v2 | accuracy: 21% -> 24% | Improved |
| use HAN+sum instead of soft attention SAN | SAN · CLEVR test | overall accuracy: 68.5 -> 89.7 | Improved |
| add relation network aggregation to HAN | HAN+sum · CLEVR test | overall accuracy: 89.7 -> 93.9 | Improved |
| scale to larger model and higher resolution (HAN+RN++) | HAN+RN+ · CLEVR test | overall accuracy: 96.9 -> 98.8 | Improved |
| use hard attention (25% cells) with relation network instead of full RN | Relation Network · CLEVR validation, 12 h training | validation accuracy: 70% -> 90% | Improved |
Closest papers by meaning
- High-Order Attention Models for Visual Question Answering (NeurIPS 2017 · similarity 0.91)
- Multi-Level Attention Networks for Visual Question Answering (CVPR 2017 · similarity 0.90)
- Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answering (CVPR 2018 · similarity 0.90)
- Structured Attentions for Visual Question Answering (ICCV 2017 · similarity 0.90)
- Hierarchical Question-Image Co-Attention for Visual Question Answering (NeurIPS 2016 · similarity 0.89)
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (ECCV 2016 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
