Research map / Vision-language and multimodal / Visual question answering
LOVA3: Learning to Visual Question Answering, Asking and Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LOVA3, which adds GenQA (question generation across generic, multi-choice, multi-turn, and grounding VQA) and EvalQA (judging QA correctness with feedback, via the new EvalQABench) training tasks to LLaVA-1.5, improving results on many multimodal benchmarks.
Problem
Current MLLMs are trained mainly to answer visual questions, leaving their ability to ask questions and to assess the correctness of question-answer pairs underexplored.
Approach
Augment instruction tuning with two extra tasks: GenQA, which trains the model to generate QA pairs from images, and EvalQA, which trains it to judge QA pairs as Yes/No with one-sentence feedback, using a new EvalQABench built from VQAv2 with MLLM-generated negatives and LLM-generated feedback.
Key results
- LOVA3-7B improves over LLaVA-1.5 on VQAv2 (80.3 vs 78.5), GQA (63.3 vs 62.0), VizWiz (53.6 vs 50.0), MME (1552.7 vs 1510.7), and MM-Vet (35.2 vs 31.2 in the main table).
- On EvalQABench test, LOVA3-7B reaches 79.58% accuracy versus 64.92% for LLaVA-1.5, with F1 79.72 vs 69.80.
- LOVA3-1.5B raises MME from 1114.7 to 1212.9 over LLaVA-Phi-1.5 with the same training recipe.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GenQA-Generic and GenQA-Grounding to VQA training (ablation) | LLaVA-1.5 / Vicuna-7B · GQA, VizWiz, ScienceQA, POPE, MME | score: GQA 62.0 -> 63.3; VizWiz 50.0 -> 53.2; ScienceQA 66.8 -> 67.4; POPE 85.9 -> 86.7; MME 1510.7 -> 1523.6 | Mixed |
| Add EvalQA to GenQA-Generic training (ablation) | LLaVA-1.5 / Vicuna-7B · GQA, VizWiz, ScienceQA, POPE, MME | score: 63.1 -> 63.7; 53.1 -> 54.4; 67.4 -> 67.0; 86.9 -> 86.9; 1550.7 -> 1520.8 | Mixed |
| Add GenQA-Generic to GenQA-Grounding and EvalQA training (ablation) | LLaVA-1.5 / Vicuna-7B · VizWiz and MME | score: 51.1 -> 53.6; 1478.7 -> 1552.7 | Helped |
| Add EvalQA to training with both GenQA groups (ablation) | LLaVA-1.5 / Vicuna-7B · GQA, VizWiz, ScienceQA, POPE, MME | score: 63.3 -> 63.3; 53.2 -> 53.6; 67.4 -> 68.0; 86.7 -> 87.4; 1523.6 -> 1552.7 | Mixed |
| Add EvalQA to VQA training (ablation) | LLaVA-1.5 / Vicuna-7B · GQA, ScienceQA, MME | score: 62.0 -> 62.8; 66.8 -> 67.8; 1510.7 -> 1535.6 | Helped |
| Train with VQA, GenQA, and EvalQA instead of VQA alone | LLaVA-1.5 / Vicuna-7B · MM-Vet | Total: +4.0 | Helped |
| Train with VQA, GenQA, and EvalQA instead of VQA alone | LLaVA-1.5 / Vicuna-7B · EvalQABench test | Accuracy: +14.66 | Helped |
| Train with VQA, GenQA, and EvalQA instead of VQA alone | LLaVA-1.5 / Vicuna-7B · POPE avg | accuracy: 85.9 -> 87.4 | Helped |
Surprising findings
- Adding only 64K EvalQA samples (one-positive-one-negative) improves GQA, ScienceQA, and MME, with EvalQA alone matching GenQA-General in several metrics.
- Grounding-based GenQA (REC/REG) data gives the smallest gains of the GenQA subsets.
Limitations
- Only 7B and 1.5B LLMs were tested; larger 13B/34B variants were not evaluated due to compute.
- GenQA and EvalQA increase training cost.
- Weak on text-centric and mathematical VQA due to limited instruction data.
Experimental setup
Datasets: VQAv2 GQA OCR-VQA LLaVA-250K A-OKVQA RefCOCO Visual Genome Counting20K LLaVA-1.5 665K instruction data
Benchmarks: VQAv2 GQA VizWiz ScienceQA POPE MME SEED-Bench MMBench LLaVA-Bench MM-Vet EvalQABench
Metrics: accuracy precision F1 score No (%)
Baselines: LLaVA-1.5 LLaVA-Phi-1.5 BLIP-2 InstructBLIP Qwen-VL MiniGPT-4 mPLUG-owl LLaMA-AdapterV2 CogVLM InternLM-XC
Builds on: LLaVA-1.5 InstructBLIP
Closest papers by meaning
- Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.89)
- Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly (CVPR 2025 · similarity 0.89)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.88)
- Right this way: Can VLMs Guide Us to See More to Answer Questions? (NeurIPS 2024 · similarity 0.88)
- UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · similarity 0.88)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.