Research map / Vision-language and multimodal / Visual question answering
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering
Mateusz Malinowski, Marcus Rohrbach, Mario Fritz
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train to predict only a single word answer | Ask Your Neurons · DAQUAR (all classes) | accuracy: 17.49 -> 19.43 | Improved |
| add visual input to question-only model (ablation) | Ask Your Neurons (single word) · DAQUAR (all classes) | accuracy: 17.15 -> 19.43 | Improved |
| add visual input to question-only model (ablation) | Ask Your Neurons · DAQUAR (reduced) | accuracy: 32.32 -> 34.68 | Improved |
| use refined model with GLOVE and ResNet-152 | Ask Your Neurons · DAQUAR (all) | accuracy: 19.43 -> 24.48 | Improved |
| add box filter smoothing of validation accuracy | Refined Ask Your Neurons · DAQUAR (all) | accuracy: 24.48 -> 25.74 | Improved |
| add GLOVE pre-trained word embedding (ablation) | LSTM question-only · VQA validation | accuracy: 47.80 -> 48.58 | Improved |
| restrict answers to the top 2000 most frequent (ablation) | LSTM question-only · VQA validation | accuracy: 48.58 -> 48.86 | Improved |
| add L2 normalisation of visual features (ablation) | Summation fusion · VQA validation | accuracy: 40.67 -> 53.27 | Improved |
| add L2 normalisation of visual features (ablation) | Concatenation fusion · VQA validation | accuracy: 47.21 -> 52.39 | Improved |
| add L2 normalisation of visual features (ablation) | Element-wise multiplication fusion · VQA validation | accuracy: 49.50 -> 52.70 | Improved |
| use ResNet-152 instead of AlexNet visual encoder | LSTM + GLOVE · VQA validation | accuracy: 53.69 -> 55.52 | Improved |
| use LSTM instead of BOW question encoder | vision and language model · VQA validation | accuracy: 53.27 -> 54.29 | Improved |
Closest papers by meaning
- Tips and Tricks for Visual Question Answering: Learnings From the 2017 Challenge (CVPR 2018 · similarity 0.92)
- An Analysis of Visual Question Answering Algorithms (ICCV 2017 · similarity 0.92)
- Image Captioning and Visual Question Answering Based on Attributes and External Knowledge (TPAMI 2017 · similarity 0.92)
- Graph-Structured Representations for Visual Question Answering (CVPR 2017 · similarity 0.92)
- Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (CVPR 2017 · similarity 0.91)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
