Research map / Vision-language and multimodal / Visual question answering
Towards Robust Visual Question Answering via Prompt-Driven Geometric Harmonization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PDGH, a prompt-driven geometric harmonization framework for robust VQA that uses MLLM-generated question-guided image captions and hypersphere geometric constraints to counter language bias and minority-class collapse.
Problem
Robust visual question answering under language bias and imbalanced answer distributions, where frequent answer classes dominate and minority classes collapse, and spurious question-answer correlations persist.
Approach
Generates question-guided image captions with an MLLM (CogVLM2) using KeyBERT keywords, projects joint and prompt-based representations onto a unit hypersphere, and adds self-orthogonality/entropy-calibration, multi-space contrastive geometric constraints, and a semantic redundancy loss.
Key results
- PDGH reaches 61.68% overall accuracy on VQA-CP v2, improving over the second-best method (CVIV, 60.08%) by 1.6 points.
- On VQA-CP v1, PDGH achieves 64.56% overall, a 1.82-point gain over the runner-up.
- On SLAKE-LB, PDGH reaches 81.21% overall versus 78.24% for RMLVQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PDGH instead of CVIV | VQA model · VQA-CP v2 | accuracy (Overall): 60.08 -> 61.68 | Helped |
| Use PDGH instead of GENB | VQA model · VQA-CP v1 | accuracy (Overall): 62.74 -> 64.56 | Helped |
| Use PDGH instead of RMLVQA | VQA model · VQA-CE | accuracy (Overall): 58.05 -> 59.10 | Helped |
| Use PDGH instead of GENB | VQA model · GQA-OOD | accuracy (All): 49.43 -> 49.56 | Helped |
| Use PDGH instead of RMLVQA | VQA model · SLAKE-LB | accuracy (Overall-CP): 78.24 -> 81.21 | Helped |
| Use QVP only, omitting OEC, CGC, and HSR (ablation) | PDGH · VQA-CP v2 | accuracy (Overall-CP): 61.69 -> 60.32 | Hurt |
| Use CGC only, omitting QVP, OEC, and HSR (ablation) | PDGH · VQA-CP v2 | accuracy (Overall-CP): 61.69 -> 60.76 | Hurt |
| Use QVP and OEC, omitting CGC and HSR (ablation) | PDGH · VQA-CP v2 | accuracy (Overall-CP): 61.69 -> 60.69 | Hurt |
| Omit HSR while retaining QVP, OEC, and CGC (ablation) | PDGH · VQA-CP v2 | accuracy (Overall-CP): 61.69 -> 61.43 | Hurt |
Limitations
- SLAKE-LB is a re-split constructed by the authors, so comparisons on it depend on their partitioning.
- Gains over strongest baselines are modest (around 0.2 to 1.8 points).
Experimental setup
Datasets: VQA-CP v2 VQA-CP v1 VQA-CE GQA-OOD SLAKE-LB
Benchmarks: VQA-CP v2 VQA-CP v1 VQA-CE GQA-OOD SLAKE-LB
Metrics: VQA accuracy overall accuracy yes/no accuracy number accuracy other accuracy
Baselines: UpDn AdvReg LMH GGE-iter COB GENB GGD PWVQA CVIV CSS RMLVQA SAN
Builds on: GGE RMLVQA Thomson problem hyperspherical energy supervised contrastive loss Barlow-twins-style redundancy reduction
Closest papers by meaning
- Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (CVPR 2017 · similarity 0.91)
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (CVPR 2018 · similarity 0.91)
- Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA (AAAI 2024 · similarity 0.91)
- REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (NeurIPS 2022 · similarity 0.90)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.90)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.