Research map / Vision-language and multimodal / Visual question answering
BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BOK-VQA, a bilingual (Korean-English) knowledge-based VQA dataset of 17.8K images/QA pairs with 280K English KG triples, and GEL-VQA, which pretrains ConvKB KG embeddings and jointly predicts triples to inject knowledge into a VQA model.
Problem
Knowledge-based VQA training data is small and mostly English, and low-resource languages such as Korean lack KG resources; the paper asks whether English KG embeddings can serve bilingual VQA and how to fuse them.
Approach
Use only English KG embeddings (ConvKB) that are language-agnostic, train a triple-prediction module from image and question to predict the needed English triple at inference, and fuse the predicted triple embedding with the multimodal VQA feature in a multitask setup.
Key results
- GEL-VQA reaches 45.08% bilingual-free accuracy on Korean-English setting versus 21.51% for the baseline, a 25.97-point gain in bilingual setting; the best variant (GEL-VQA+TF+ATTN) reaches 55.48% on Korean.
- The ideal oracle GEL-VQA(Ideal) reaches up to 72.25% on Korean, showing the dataset depends on external knowledge.
- Triple prediction overall accuracy is about 47-50% across bilingual, English, and Korean settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use predicted external triples with GEL-VQA instead of omitting external knowledge | BOK-VQA · Bilingual | accuracy: 21.51$\pm$1.81 -> 45.08$\pm$0.94 | Helped |
| Use predicted external triples with GEL-VQA instead of omitting external knowledge | BOK-VQA · English | accuracy: 21.90$\pm$2.34 -> 47.83$\pm$0.56 | Helped |
| Use predicted external triples with GEL-VQA instead of omitting external knowledge | BOK-VQA · Korean | accuracy: 21.16$\pm$1.50 -> 50.30$\pm$2.24 | Helped |
| Use known triples rather than predicting them | GEL-VQA · BOK-VQA bilingual | accuracy: 45.08$\pm$0.94 -> 66.01$\pm$1.83 | Helped |
| Use ground-truth triples during training (ablation) | GEL-VQA · BOK-VQA bilingual | accuracy: 45.08$\pm$0.94 -> 48.07$\pm1$.33 | Helped |
| Use ground-truth triples during training (ablation) | GEL-VQA · BOK-VQA English | accuracy: 47.83$\pm$0.56 -> 51.64$\pm$0.88 | Helped |
| Use ground-truth triples during training (ablation) | GEL-VQA · BOK-VQA Korean | accuracy: 50.30$\pm$2.24 -> 53.40$\pm$2.73 | Helped |
| Add multi-head attention to triple embeddings after teacher forcing (ablation) | GEL-VQA + TF · BOK-VQA bilingual | accuracy: 48.07$\pm1$.33 -> 48.11$\pm$1.50 | No clear effect |
| Add multi-head attention to triple embeddings after teacher forcing (ablation) | GEL-VQA + TF · BOK-VQA English | accuracy: 51.64$\pm$0.88 -> 50.74$\pm$0.90 | Hurt |
| Add multi-head attention to triple embeddings after teacher forcing (ablation) | GEL-VQA + TF · BOK-VQA Korean | accuracy: 53.40$\pm$2.73 -> 55.48$\pm$1.89 | Helped |
| Use TransE rather than ConvKB for KGE pretraining (ablation) | GEL-VQA · BOK-VQA | accuracy: 45.08$\pm$0.94 -> 51.22$\pm$1.02 | Helped |
| Replace original questions with noun hypernym transformations (ablation) | GEL-VQA · BOK-VQA bilingual | accuracy: 45.08$\pm$0.94 -> 36.24$\pm$0.76 | Hurt |
Surprising findings
- Korean-question performance was on average about 3.7% higher than English despite using only an English KB.
- Accuracy under ConvKB KGE was lower than TransE/TorusE in the GEL-VQA setting even though ConvKB was the default embedding.
Limitations
- Predicts and uses only one knowledge triple per question, so multi-hop or multi-fact reasoning is not handled.
- English questions were machine-translated from Korean and only partly revised, which may limit sentence-structure standardization.
- Triple prediction accuracy is modest (about 47-50% overall).
Experimental setup
Datasets: BOK-VQA ConceptNet DBpedia ImageNet FB15K WordNet VQA v2.0
Benchmarks: BOK-VQA
Metrics: accuracy triple prediction accuracy Hit@10
Baselines: baseline VQA (XLM-RoBERTa + ResNet50) GEL-VQA(Ideal) TransE TorusE HolE DistMult
Builds on: ConvKB XLM-RoBERTa ResNet
Closest papers by meaning
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.92)
- Prompting Large Language Models With Answer Heuristics for Knowledge-Based Visual Question Answering (CVPR 2023 · similarity 0.92)
- REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (NeurIPS 2022 · similarity 0.91)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.90)
- Multimodal Commonsense Knowledge Distillation for Visual Question Answering (Student Abstract) (AAAI 2025 · similarity 0.90)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.