Research map / Vision-language and multimodal / Visual question answering
Learning to Reason: End-To-End Module Networks for Visual Question Answering
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Kate Saenko
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add second-stage policy search (RL) after expert cloning (ablation) | N2NMN · CLEVR test | overall accuracy: 78.9 -> 83.7 | Improved |
| add second-stage policy search (RL) after expert cloning (ablation) | N2NMN · CLEVR test | Compare Attribute color accuracy: 52.5 -> 82.8 | Improved |
| add behavioral cloning from expert layouts before policy search (ablation) | N2NMN · CLEVR test | overall accuracy: 69.0 -> 83.7 | Improved |
| clone expert layouts instead of pure policy search from scratch (ablation) | N2NMN · CLEVR test | overall accuracy: 69.0 -> 78.9 | Improved |
| add behavioral cloning from expert layouts instead of policy search from scratch (ablation) | N2NMN · SHAPES | accuracy: 96.19% -> 100.00% | Improved |
| predict layouts with a learned policy instead of a ground-truth parse at test time | N2NMN vs NMN · SHAPES | accuracy: 90.80% -> 100.00% | Improved |
| add policy search after cloning on VQA (ablation) | N2NMN ResNet-152 · VQA test-dev | accuracy: 64.2 -> 64.9 | Improved |
| use ResNet-152 features instead of LRCN VGG-16 (ablation) | N2NMN cloning expert · VQA test-dev | accuracy: 61.9 -> 64.2 | Improved |
| use learned layout with soft-attention module parameters instead of parser-based NMN, same VGG-16 features | N2NMN · VQA test-dev | accuracy: 57.9 -> 61.9 | Improved |
| N2NMN versus MCB on same ResNet-152 features | N2NMN · VQA test-dev | accuracy: 64.7 -> 64.9 | No clear effect |
Closest papers by meaning
- Neural Module Networks (CVPR 2016 · similarity 0.92)
- Neural Module Networks for Reasoning over Text (ICLR 2020 · similarity 0.91)
- Visual Question Reasoning on General Dependency Tree (CVPR 2018 · similarity 0.90)
- Transformer Module Networks for Systematic Generalization in Visual Question Answering (TPAMI 2024 · similarity 0.89)
- Dynamic Memory Networks for Visual and Textual Question Answering (ICML 2016 · similarity 0.88)
- Detection-Based Intermediate Supervision for Visual Question Answering (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
