Research map / Vision-language and multimodal / Visual question answering
FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FRAMES-VQA, a benchmark of ten VQA datasets grouped into ID, near-OOD and far-OOD multi-modal shifts, and compares robust fine-tuning methods on PaliGemma-3B with LoRA, using Mahalanobis-distance shift scores and modality-importance attention analysis.
Problem
Existing robust fine-tuning evaluations for vision-language models are mostly unimodal or cover only particular OOD types, giving limited insight into multi-modal distribution shifts in VQA.
Approach
Builds a VQA-specific robust fine-tuning benchmark that trains on VQAv2 and evaluates on image, question, answer, multi-modal, adversarial and far-OOD shifts, quantifies shifts with Mahalanobis distance over uni-modal and joint embeddings, and relates shift scores to intra- vs inter-modality attention.
Key results
- SPD (LoRA) achieves the highest ID (87.39) and near-OOD average (75.88) VQA accuracy among the compared robust fine-tuning methods.
- FTP (LoRA) achieves the best far-OOD average (51.07) across TextVQA, VizWiz and OK-VQA, while underfitting ID data (81.77 vs 86.29 for vanilla FT).
- Vanilla fine-tuning does not degrade OOD performance relative to zero-shot in VQA, unlike reported image-classification results; WiSE-FT lowers both ID and OOD accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply vanilla LoRA fine-tuning instead of zero-shot evaluation | PaliGemma-3B · VQAv2 val | accuracy: 54.42 -> 86.29 | Helped |
| Apply vanilla LoRA fine-tuning instead of zero-shot evaluation | PaliGemma-3B · Near OOD Avg. | accuracy: 45.70 -> 75.08 | Helped |
| Apply vanilla LoRA fine-tuning instead of zero-shot evaluation | PaliGemma-3B · Far OOD Avg. | accuracy: 20.10 -> 37.77 | Helped |
| Blend pretrained and fine-tuned weights with WiSE-FT | PaliGemma-3B fine-tuning · VQAv2 val | accuracy: 86.29 -> 71.36 | Hurt |
| Blend pretrained and fine-tuned weights with WiSE-FT | PaliGemma-3B fine-tuning · Near OOD Avg. | accuracy: 75.08 -> 63.27 | Hurt |
| Use SPD selective projection fine-tuning | PaliGemma-3B · VQAv2 val | accuracy: 87.39 | Helped |
| Use SPD selective projection fine-tuning | PaliGemma-3B · Near OOD Avg. | accuracy: 75.88 | Helped |
| Use SPD selective projection fine-tuning | PaliGemma-3B · OOD Avg. | accuracy: 63.55 | Helped |
| Use FTP projection fine-tuning | PaliGemma-3B · Far OOD Avg. | accuracy: 41.95 | Helped |
| Use FTP projection fine-tuning | PaliGemma-3B · VQAv2 val | accuracy: 86.29 -> 81.77 | Hurt |
| Use FTP projection fine-tuning | PaliGemma-3B · Near OOD Avg. | accuracy: 75.08 -> 72.29 | Hurt |
Surprising findings
- Vanilla fine-tuning improves over zero-shot on OOD VQA datasets, contrary to prior image-classification findings of OOD degradation.
- WiSE-FT reduces both ID and OOD performance in the VQA setting, making it less effective than vanilla fine-tuning.
Limitations
- Evaluation uses only PaliGemma-3B with LoRA as the main backbone; LLaVA and full fine-tuning results are only in the supplement.
- Mahalanobis shift scores are computed from embeddings of specific backbones, so shift quantification depends on the chosen encoder.
- CV-VQA is a small outlier with low shift scores, possibly due to sample size.
Experimental setup
Datasets: VQAv2 IV-VQA CV-VQA VQA-Rephrasings VQA-CP VQA-CE AdVQA TextVQA VizWiz OK-VQAv2
Benchmarks: VQAv2 IV-VQA CV-VQA VQA-Rephrasings VQA-CP VQA-CE AdVQA TextVQA VizWiz OK-VQAv2
Metrics: VQA accuracy Mahalanobis distance Pearson correlation modality importance ratio
Baselines: Zero-shot Vanilla FT Linear Probing LP-FT WiSE-FT FTP SPD
Builds on: TPGM FTP SPD WiSE-FT LP-FT
Closest papers by meaning
- Cycle-Consistency for Robust Visual Question Answering (CVPR 2019 · similarity 0.92)
- Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (CVPR 2017 · similarity 0.92)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.92)
- Knowledge Exchange with Confidence: Cost-Effective LLM Integration for Reliable and Efficient Visual Question Answering (ICLR 2026 · similarity 0.92)
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (CVPR 2018 · similarity 0.92)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.