Research map / Vision-language and multimodal / Visual question answering
VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VTQA, a 23,781-question open-ended QA dataset over COCO image-text pairs requiring entity alignment and multi-hop image-text reasoning, plus a baseline KECMRN that outperforms other VQA models and MLLMs on it.
Problem
Existing VQA benchmarks mostly test simple visual perception, and multimodal QA datasets either rely on extracted/automatically generated QA or simplify image reasoning, so they do not measure cross-media entity alignment and multi-step reasoning over image and text.
Approach
Human-annotated long text paired with COCO images, with questions filtered so they cannot be answered from image or text alone; a baseline (KECMRN) iteratively extracts question-relevant key entities from text and image and applies cross-media reasoning layers built on MCAN-style attention units.
Key results
- KECMRN achieves 57.95% EM and 74.99% YN accuracy on the English VTQA test set, the best among compared models (GPT4V 48.93% EM).
- Zero-shot MLLMs and pretrained VLMs (LLaVA, GPT4V, MMICL, BEiT3, VLMo) drop substantially on VTQA, with BEiT3-base reaching 41.02% EM.
- Ablation shows the KEE layer adds about 0.46 EM and the CMR layer about 0.32 EM over the Trans-CA variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use KECMRN instead of Trans-CA | VTQA · VTQA English test | EM: 57.38 -> 57.95 | Helped |
| use KECMRN instead of Trans-CA | VTQA · VTQA English test | YN-ACC: 73.57 -> 74.99 | Helped |
| use KECMRN instead of GPT4V | VTQA · VTQA English test | E-F1: 70.24 -> 68.91 | Hurt |
| use KECMRN instead of GPT4V | VTQA · VTQA English test | G-F1: 48.37 -> 44.25 | Hurt |
| use KECMRN instead of MCAN-grid | VTQA · VTQA Chinese test | EM: 50.22 -> 51.32 | Helped |
| use KECMRN instead of Trans-CA | VTQA · VTQA Chinese test | YN-ACC: 74.27 -> 77.59 | Helped |
| use KECMRN instead of Trans-CA | VTQA · VTQA Chinese test | E-F1: 59.63 -> 60.52 | Helped |
| use KECMRN instead of Trans-CA | VTQA · VTQA Chinese test | G-F1: 47.45 -> 51.11 | Helped |
| use an LSTM instead of a Transformer text encoder (ablation) | KECMRN · VTQA English test | EM: 55.45; 55.07; 54.07 -> 57.95 | Helped |
| use an LSTM instead of a Transformer text encoder (ablation) | KECMRN · VTQA English test | YN-ACC: 72.43; 70.98; 69.43 -> 74.99 | Helped |
| use an LSTM instead of a Transformer text encoder (ablation) | KECMRN · VTQA English test | E-F1: 65.35; 65.7; 64.83 -> 68.91 | Helped |
| use an LSTM instead of a Transformer text encoder (ablation) | KECMRN · VTQA English test | G-F1: 42.27; 43.58; 43.91 -> 44.25 | Helped |
Surprising findings
- Deeper transformer text encoders performed worse than a one-layer LSTM encoder, attributed to limited training data and model complexity.
- Large pretrained multimodal models (GPT4V, LLaVA, BEiT3) underperform simpler task-specific attention models trained from scratch on VTQA.
Limitations
- Dataset is relatively small (about 10k images, 23.8k questions), limiting training of large models.
- Exact match metric is limited for open-ended answers; the authors call for better evaluation metrics.
- Annotations are in Chinese and translated to English, and no reasoning-chain annotations are provided.
Experimental setup
Datasets: VTQA VQAv2
Benchmarks: VTQA
Metrics: exact match YN accuracy F1
Baselines: LLaVA GPT-4V MMICL BEiT3-base VLMo-base MCAN-region MCAN-grid Trans-CA Trans-SA
Builds on: MCAN Transformer BEiT-3 VLMo
Closest papers by meaning
- Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (CVPR 2017 · similarity 0.93)
- KVQA: Knowledge-Aware Visual Question Answering (AAAI 2019 · similarity 0.93)
- Visual Question Answering as Reading Comprehension (CVPR 2019 · similarity 0.92)
- Answer Them All! Toward Universal Visual Question Answering Models (CVPR 2019 · similarity 0.92)
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (CVPR 2019 · similarity 0.92)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.