Research map / Vision-language and multimodal / Visual question answering
From Strings to Things: Knowledge-Enabled VQA Model That Can Read and Reason
Ajeet Kumar Singh, Anand Mishra, Shashank Shekhar, Anirban Chakraborty
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual content proposals to text-only GGNN (photoOCR-1) (ablation) | Full model · text-KVQA (scene) | VQA accuracy: 48.9 -> 52.2 | Improved |
| add visual content proposals to text-only GGNN (photoOCR-1) (ablation) | Full model · text-KVQA (book) | VQA accuracy: 55.0 -> 62.7 | Improved |
| add visual content proposals to text-only GGNN (photoOCR-1) (ablation) | Full model · text-KVQA (movie) | VQA accuracy: 41.4 -> 45.2 | Improved |
| add visual content proposals to text-only GGNN (photoOCR-2) (ablation) | Full model · text-KVQA (scene) | VQA accuracy: 52.2 -> 54.5 | Improved |
| add visual content proposals to text-only GGNN (photoOCR-2) (ablation) | Full model · text-KVQA (book) | VQA accuracy: 48.6 -> 49.8 | Improved |
| add visual content proposals to text-only GGNN (photoOCR-2) (ablation) | Full model · text-KVQA (movie) | VQA accuracy: 20.5 -> 23.0 | Improved |
| add word proposals to vision-only model (ablation) | Vision only · text-KVQA (scene) | VQA accuracy: 21.8 -> 52.2 | Improved |
| replace memory network with GGNN reasoning (photoOCR-1) | Full model · text-KVQA (book) | VQA accuracy: 57.2 -> 62.7 | Improved |
| replace memory network with GGNN reasoning (photoOCR-2) | Full model · text-KVQA (movie) | VQA accuracy: 22.2 -> 23.0 | Improved |
| replace noisy OCR with ideal text recognition | Full model · text-KVQA (scene) | VQA accuracy: 54.5 -> 80.1 | Improved |
| add visual proposals and question to word proposals in fact retrieval (photoOCR2) (ablation) | Fusion module · text-KVQA (scene) | fact recall@top-100: 59.9 -> 60.7 | Improved |
| add visual proposals and question to word proposals in fact retrieval (photoOCR1) (ablation) | Fusion module · text-KVQA (scene) | fact recall@top-100: 55.8 -> 58.9 | Improved |
Closest papers by meaning
- Scene Text Visual Question Answering (ICCV 2019 · similarity 0.92)
- Visual Question Answering as Reading Comprehension (CVPR 2019 · similarity 0.92)
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (CVPR 2019 · similarity 0.91)
- KVQA: Knowledge-Aware Visual Question Answering (AAAI 2019 · similarity 0.91)
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.90)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
