Research map / Vision-language and multimodal / Visual question answering
Explicitly Guided Difficulty-Controllable Visual Question Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines visual question difficulty as the number of reasoning steps and proposes MultiStepGen, which builds GQA-derived reasoning chains from scene graphs and progressively rewrites questions to add hops for difficulty-controllable visual question generation.
Problem
Visual question generation has focused on fluency and relevance while neglecting controllable question difficulty; prior difficulty definitions were binary (easy/hard) and tied to specific VQA models.
Approach
Define difficulty as the number of reasoning steps (hops) in a scene-graph relationship chain, and generate questions step by step: a 1-hop question is produced first, then iteratively rewritten to include additional sub-chains, with GPT-2 as the generator and CLIP patch features fused by attention.
Key results
- On the 1-hop DVQA split, MultiStepGen reaches BLEU-4 32.26 and CIDEr 3.51, versus 31.66 BLEU-4 for ChatGPT and 17.72 for MS-VQG.
- On the 2-hop split, MultiStepGen reaches BLEU-4 18.01 and CIDEr 2.11, outperforming VisualBERT (14.36 BLEU-4) and ChatGPT (7.23 BLEU-4).
- Human evaluation gives MultiStepGen a Difficulty score of 1.98 on 2-hop questions versus 1.44 for VisualBERT, with the 1-hop/2-hop difficulty gap larger than for VisualBERT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MultiStepGen instead of ChatGPT | visual question generation · DVQA 1-hop | BLEU 4: 31.66 -> 32.26 | Helped |
| use MultiStepGen instead of VISUAL-BERT | visual question generation · DVQA 2-hop | BLEU 4: 14.36 -> 18.01 | Helped |
| add visual feature extractor (ablation) | MultiStepGen · DVQA 1-hop | BLEU 4: 25.79 -> 32.26 | Helped |
| add controllable rewriting module (ablation) | MultiStepGen · DVQA 1-hop | BLEU 4: 26.13 -> 32.26 | Helped |
| include answer information (ablation) | MultiStepGen · DVQA 1-hop | BLEU 4: 30.61 -> 32.26 | Helped |
| add visual feature extractor (ablation) | MultiStepGen · DVQA 2-hop | BLEU 4: 15.18 -> 18.01 | Helped |
| add controllable rewriting module (ablation) | MultiStepGen · DVQA 2-hop | BLEU 4: 14.79 -> 18.01 | Helped |
| include answer information (ablation) | MultiStepGen · DVQA 2-hop | BLEU 4: 16.23 -> 18.01 | Helped |
| increase the target reasoning depth from 1-hop to 2-hop | MultiStepGen · DVQA | human-rated Difficulty: 1.02 -> 1.98 | Helped |
Surprising findings
- ChatGPT, which helped construct some 1-hop questions, drops sharply on 2-hop generation (BLEU-4 7.23), performing worse than several non-LLM baselines.
Limitations
- Only 1-hop and 2-hop question pairs are used for training; multi-hop generalization is validated on a small set of 100 questions.
- Difficulty is measured via reasoning-step count and VQA model accuracy, and the dataset relies on GQA 'relate' samples and ChatGPT-generated 1-hop questions with manual checks.
- Human evaluation uses five volunteers on 200 samples.
Experimental setup
Datasets: GQA Visual Genome scene graphs
Benchmarks: DVQA
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-L METEOR CIDEr human fluency/relevance/difficulty/answerability VQA accuracy of Up-Down, DFAF, Counter, Graph
Baselines: GRNN IM-VQG Radial-GCN MOAG MS-VQG ChatGPT VisualBERT DGN
Builds on: GQA Visual Genome GPT-2 CLIP DGN
Closest papers by meaning
- ConVQG: Contrastive Visual Question Generation with Multimodal Guidance (AAAI 2024 · similarity 0.91)
- Understanding Complexity in VideoQA via Visual Program Generation (ICML 2025 · similarity 0.89)
- Image Content Generation with Causal Reasoning (AAAI 2024 · similarity 0.87)
- How Teachers Can Use Large Language Models and Bloom’s Taxonomy to Create Educational Quizzes (AAAI 2024 · similarity 0.86)
- Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation (CVPR 2025 · similarity 0.86)
- ViUniT: Visual Unit Tests for More Robust Visual Programming (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.