Research map / Vision-language and multimodal / Visual question answering
Transformer Module Networks for Systematic Generalization in Visual Question Answering
Moyuru Yamada, Vanessa D’Amario, Kentaro Takemoto, Xavier Boix, Tomotake Sasaki
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| modularize Transformer with question-specific module composition (TMN-Tree vs Transformer) | Transformer · CLOSURE | accuracy: 57.4 -> 95.4 | Improved |
| use TMN-Stack instead of Transformer | Transformer · CLOSURE | accuracy: 57.4 -> 90.9 | Improved |
| use TMN-Tree vs Transformer on GQA-SGL | Transformer · GQA-SGL | accuracy: 47.7 -> 53.7 | Improved |
| use TMN on novel visual attribute combinations | Transformer · CLEVR-CoGenT condition B | accuracy: 78.9 -> 80.6 | No clear effect |
| add program-related (PR) tokens to Transformer | Transformer · CLOSURE | accuracy: 57.4 -> 64.5 | Improved |
| add specialization of modules (SoM) to variable-layer split-token Transformer (ablation) | Transformer w/ PR + VL + ST · CLOSURE | accuracy: 43.6 -> 90.9 | Improved |
| add variable number of layers (VL) without specialization (ablation) | Transformer w/ PR · CLOSURE | accuracy: 64.5 -> 60.2 | Hurt |
| add split tokens (ST) without specialization (ablation) | Transformer w/ PR + VL · CLOSURE | accuracy: 60.2 -> 43.6 | Hurt |
| assign modules to individual sub-tasks vs order-based modules (ablation) | TMN-Stack · CLOSURE | accuracy: 68.4 -> 90.9 | Improved |
| group semantically similar sub-tasks into shared modules (ablation) | TMN-Stack · CLOSURE | accuracy: 90.9 -> 93.7 | Improved |
| use object detector pre-trained on Visual Genome regional features (ablation) | TMN-Tree · CLOSURE | accuracy: 95.4 -> 96.4 | Improved |
| use object detector pre-trained on Visual Genome (ablation) | Transformer · CLEVR-CoGenT condition B | accuracy: 78.9 -> 87.5 | Improved |
Closest papers by meaning
- How Modular should Neural Module Networks Be for Systematic Generalization? (NeurIPS 2021 · similarity 0.93)
- Neural Module Networks (CVPR 2016 · similarity 0.91)
- Robust Visual Reasoning via Language Guided Neural Module Networks (NeurIPS 2021 · similarity 0.90)
- Learning to Reason: End-To-End Module Networks for Visual Question Answering (ICCV 2017 · similarity 0.89)
- Deep Modular Co-Attention Networks for Visual Question Answering (CVPR 2019 · similarity 0.89)
- Ask Your Neurons: A Deep Learning Approach to Visual Question Answering (IJCV 2017 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
