Research map / Vision-language and multimodal / Visual question answering
When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the OVVQA benchmark (base vs. novel answer classes built from VQA v2, GQA, OK-VQA) and a model-agnostic front-door-adjustment Causal Adapter plus an adaptive transfer loss, plugged into VL-T5 and VL-BART.
Problem
Closed-set VQA benchmarks fail to evaluate generalization to unseen, rare answers; fine-tuned VQA models improve on common answers but degrade on novel ones due to a bias the authors term canonical class overfitting.
Approach
Formulate OVVQA with a causal graph where a confounder links image to seen-class labels, and apply front-door adjustment inside a trainable adapter between frozen encoder and decoder, combined with an adaptive loss that nonlinearly fuses auxiliary pretext losses (ITM, ITC, MLM) via a Transformer.
Key results
- On OV-VQAv2 with VL-T5, novel-class accuracy rose from 6.02% (baseline) to 6.7% with the causal adapter per the text; ablation shows full model Novel 32.69 vs baseline 28.29 on the 'novel' column of Table 4.
- On OV-GQA with VL-T5, novel accuracy improved from 5.21% to 10.88% (text), while full fine-tuning dropped novel accuracy to 12.46.
- On OV-OKVQA with VL-T5, harmonic mean rose from 32.24 (baseline) to 36.12 with the proposed method, with novel accuracy 29.10 vs 27.91.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Causal Adapter and adaptive transfer loss | VL-T5 · OV-VQAv2 | H: 40.37 -> 44.44 | Helped |
| add Causal Adapter and adaptive transfer loss | VL-T5 · OV-GQA | H: 30.92 -> 33.64 | Helped |
| add Causal Adapter and adaptive transfer loss | VL-T5 · OV-OKVQA | H: 23.33 -> 36.12 | Helped |
| replace the standard Transformer adapter with causal attention (ablation) | VL-T5 · OV-VQAv2 | Novel: 25.75 -> 30.15 | Helped |
| replace the standard Transformer adapter with causal attention (ablation) | VL-BART · OV-VQAv2 | Novel: 25.18 -> 28.48 | Helped |
| add an adaptive transfer loss to the baseline (ablation) | VL-T5 · OV-VQAv2 | Novel: 28.29 -> 28.97 | Helped |
| add an adaptive transfer loss to the baseline (ablation) | VL-BART · OV-VQAv2 | Novel: 26.85 -> 27.01 | Helped |
| use a standard Transformer adapter instead of causal attention (ablation) | VL-T5 · OV-VQAv2 | Novel: 28.29 -> 25.75 | Hurt |
| use a standard Transformer adapter instead of causal attention (ablation) | VL-BART · OV-VQAv2 | Novel: 26.85 -> 25.18 | Hurt |
| vary the number of k-means cluster centers K (ablation) | Causal Adapter · OV-VQAv2 | accuracy | Mixed |
Surprising findings
- Standard fine-tuning (Full-Finetune, CoOp, CLIP-Adapter) improves base-class accuracy but degrades novel-class accuracy, which the authors attribute to canonical class overfitting bias.
- A standard transformer adapter (w/ATT-Adapter) lowers novel-class accuracy relative to the baseline on OV-VQAv2 (e.g., 25.75 vs 28.29 with VL-T5).
Limitations
- Novel-class accuracy remains low in absolute terms (e.g., about 6-11% on OV-VQAv2/GQA for VL-T5 baseline variants).
- Evaluation is limited to VQA-style answer-classification datasets derived from closed-set VQA data; generalization to other multimodal tasks is untested.
Experimental setup
Datasets: VQA v2 GQA OK-VQA
Benchmarks: OV-VQAv2 OV-GQA OV-OKVQA
Metrics: base accuracy novel accuracy arithmetic mean harmonic mean
Baselines: Full-Finetune CoOp CLIP-Adapter LXMERT ALBEF RepARe w/ ATT-Adapter
Builds on: VL-T5 VL-BART CLIP-Adapter CoOp front-door adjustment
Closest papers by meaning
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (CVPR 2018 · similarity 0.94)
- Answer-Type Prediction for Visual Question Answering (CVPR 2016 · similarity 0.94)
- Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (CVPR 2017 · similarity 0.94)
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (CVPR 2019 · similarity 0.93)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.92)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.