Research map / Vision-language and multimodal / Visual question answering
Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
Qi Wu, Chunhua Shen, Peng Wang, Anthony Dick, Anton van den Hengel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CNN image feature with predicted attribute vector as LSTM input | VggNet+LSTM captioner · MS COCO captioning | CIDEr: 0.56 -> 0.94 | Improved |
| add region-based attribute prediction instead of global-image prediction (ablation) | Att+LSTM captioner · MS COCO captioning | CIDEr: 0.83 -> 0.94 | Improved |
| add region CNN attribute prediction instead of linear SVM (ablation) | Att+LSTM captioner · MS COCO captioning | CIDEr: 0.82 -> 0.94 | Improved |
| use attributes instead of fine-tuned VggNet features | VggNet+ft+LSTM · MS COCO captioning | CIDEr: 0.73 -> 0.94 | Improved |
| add ground-truth attributes (oracle) (ablation) | Att+LSTM captioner · MS COCO captioning | CIDEr: 0.94 -> 1.07 | Improved |
| add image caption representation to attributes (ablation) | Att-LSTM · VQA validation | overall accuracy %: 51.60 -> 55.04 | Improved |
| add external knowledge to attributes (ablation) | Att-LSTM · VQA validation | overall accuracy %: 51.60 -> 53.79 | Improved |
| add external knowledge to attributes and captions (ablation) | Att+Cap-LSTM · VQA validation | overall accuracy %: 55.04 -> 55.96 | Improved |
| select knowledge guided by the question (ablation) | Att+Cap+Know-LSTM · VQA validation | overall accuracy %: 55.96 -> 56.17 | Improved |
| add external knowledge on why questions (ablation) | Att-LSTM · VQA validation, why questions | accuracy %: 7.77 -> 11.88 | Improved |
| add question-selected knowledge on why questions (ablation) | A+C+K-LSTM · VQA validation, why questions | accuracy %: 13.53 -> 13.76 | Improved |
| add question-selected knowledge on Toronto COCO-QA (ablation) | Att+Cap+Know-LSTM · Toronto COCO-QA | accuracy %: 69.73 -> 70.98 | Improved |
Closest papers by meaning
- Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge From External Sources (CVPR 2016 · similarity 0.94)
- What Value Do Explicit High Level Concepts Have in Vision to Language Problems? (CVPR 2016 · similarity 0.93)
- Multi-Level Attention Networks for Visual Question Answering (CVPR 2017 · similarity 0.93)
- An Analysis of Visual Question Answering Algorithms (ICCV 2017 · similarity 0.93)
- Visual Question Answering with Question Representation Update (QRU) (NeurIPS 2016 · similarity 0.92)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
