Research map / Vision-language and multimodal / Visual question answering
ConVQG: Contrastive Visual Question Generation with Multimodal Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConVQG, a BLIP-based visual question generator trained with dual image/text contrastive losses that push generated questions away from image-only and text-only question embeddings, evaluated on K-VQG, VQA 2.0 and VQG-COCO.
Problem
Text-guided visual question generation often ignores either the image grounding or the textual constraint (answer, knowledge triplet, caption), producing generic or weakly grounded questions.
Approach
A dual contrastive objective uses frozen image-only and text-only question generation modules as negatives, so the multimodal question embedding must be close to the ground truth and far from single-modality embeddings, enforcing both image grounding and text constraint satisfaction.
Key results
- On K-VQG with knowledge triplets, ConVQG_IT reaches BLEU-4 20.01 and CIDEr 1.53, versus 18.84 and 1.31 for K-VQG.
- On VQA 2.0 small, ConVQG_IT reaches BLEU-4 33.1 and CIDEr 2.79, versus 28.1 and 2.39 for the previous best MOAG.
- In human preference on 500 K-VQG pairs, ConVQG_IT was preferred 236 times versus 183 for the non-contrastive baseline, with 81 'similar' votes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add both image and text contrastive objectives (ablation) | ConVQG baseline · K-VQG (answer) | BLEU-4: 12.95 -> 14.30 | Helped |
| Add both image and text contrastive objectives (ablation) | ConVQG baseline · K-VQG (answer) | CIDEr: 0.64 -> 0.78 | Helped |
| Add both image and text contrastive objectives (ablation) | ConVQG baseline · K-VQG (knowledge triplet) | BLEU-4: 18.33 -> 20.01 | Helped |
| Add both image and text contrastive objectives | ConVQG baseline · FVQA transfer | METEOR: 13.78 -> 13.77 | No clear effect |
| Use image contrastive objective only (ablation) | ConVQG baseline · K-VQG (answer) | BLEU-4: 12.95 -> 13.95 | Helped |
| Use text contrastive objective only (ablation) | ConVQG baseline · K-VQG (answer) | BLEU-4: 12.95 -> 13.97 | Helped |
| Set alpha to 0.5 instead of 0.2 (ablation) | ConVQG_IT · K-VQG (knowledge triplet) | BLEU-4: 20.01 -> 19.90 | Hurt |
| Set alpha to 0.8 instead of 0.2 (ablation) | ConVQG_IT · K-VQG (knowledge triplet) | BLEU-4: 20.01 -> 19.79 | Hurt |
| Use fixed beta 10 instead of linearly changing beta (ablation) | ConVQG_IT · K-VQG (knowledge triplet) | BLEU-4: 20.01 -> 19.80 | Hurt |
| Use fixed beta 100 instead of linearly changing beta (ablation) | ConVQG_IT · K-VQG (knowledge triplet) | BLEU-4: 20.01 -> 19.74 | Hurt |
| Set margin m to 0.2 instead of 0.5 (ablation) | ConVQG_IT · K-VQG (knowledge triplet) | BLEU-4: 20.01 -> 19.89 | Hurt |
| Set margin m to 0.8 instead of 0.5 (ablation) | ConVQG_IT · K-VQG (knowledge triplet) | BLEU-4: 20.01 -> 19.68 | Hurt |
Surprising findings
- Reported gains over the baseline are small on K-VQG (e.g., CIDEr 1.31 to 1.53 with the full model and only about 0.14 CIDEr gain on answer constraints), and the METEOR score on K-VQG with triplets is slightly lower than K-VQG's.
Limitations
- Evaluation relies mainly on n-gram metrics (BLEU, METEOR, CIDEr) against single reference questions.
- FVQA transfer results are weak and the authors note FVQA questions are poorly grounded to images.
- Human evaluation covers only 500 K-VQG pairs and compares only against the non-contrastive baseline.
Experimental setup
Datasets: K-VQG VQA 2.0 VQG-COCO FVQA
Benchmarks: K-VQG VQA 2.0 small VQA 2.0 large VQG-COCO FVQA
Metrics: BLEU-4 BLEU-1 METEOR CIDEr ROUGE_L
Baselines: K-VQG IM-VQG IVQA iQAN Radial-GCN MOAG C3VQG MDN MC-BMN I2Q Creative SAT
Builds on: BLIP sentence-BERT K-VQG IM-VQG
Closest papers by meaning
- Explicitly Guided Difficulty-Controllable Visual Question Generation (AAAI 2025 · similarity 0.91)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.89)
- Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering (NeurIPS 2025 · similarity 0.89)
- Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation (ICLR 2024 · similarity 0.89)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.89)
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.