Research map / Vision-language and multimodal / Visual question answering
Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MCLE, a GPT-2-based vision-language model for VQA natural language explanation that generates explanation-then-answer chains and is trained with semantic-, image-, and instance-level contrastive losses over factual and counterfactual samples.
Problem
Post-hoc and self-rationalizing VQA natural language explanation (VQA-NLE) models often produce explanations that do not logically lead to the answer, contradict image facts, or ignore small semantic perturbations, reducing faithfulness.
Approach
Treats VQA-NLE as chain-of-thought generation (explanation before answer) and adds three contrastive objectives: SemanticCL (explanation+question anchor vs. negative answers), ImageCL (explanation+answer anchor vs. counterfactual images retrieved by question similarity), and InstanceCL (vs. gradient-based masked object/word counterfactuals).
Key results
- On VQA-X (unfiltered), MCLE reaches CIDEr 106.7, SPICE 23.9, answer accuracy 87.7, and human score 80.8, beating S3C (104.4, 21.5, 85.6, 77.4).
- On A-OKVQA (unfiltered), MCLE reaches CIDEr 76.6 and accuracy 34.7 vs. S3C's 74.4 and 33.5.
- Human evaluation reduces the share of unqualified explanations by type: deductive unsatisfiability from 18.9% to 17.1%, factual inconsistency from 17.3% to 15.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MCLE instead of the strongest baseline, S³C | VQA-X · VQA-X unfiltered | Human evaluation: 77.4 -> 80.8 | Helped |
| Use MCLE instead of the strongest baseline, S³C | A-OKVQA · A-OKVQA unfiltered | CIDEr: 74.4 -> 76.6 | Helped |
| Add chain-of-thought generation (ablation) | MCLE · VQA-X | CIDEr: CIDEr reduces by 1.2% on VQA-X | Helped |
| Add SemanticCL (ablation) | MCLE · VQA-X | SPICE: SPICE reduces by 1.6% on VQA-X | Helped |
| Add ImageCL (ablation) | MCLE · VQA-X | ROUGE-L: ROUGE-L reduces by 3.6% on VQA-X | Helped |
| Add InstanceCL (ablation) | MCLE · VQA-X | ROUGE-L: ROUGE-L reduces by 2.4% on VQA-X | Helped |
| Add COT and all contrastive-learning components (ablation) | MCLE · VQA-X | CIDEr: CIDEr reduces by 8.6% on VQA-X | Helped |
Limitations
- Evaluated only on two VQA-NLE benchmarks with a single GPT-2 backbone.
- Human evaluation uses only three evaluators and small reported gains on some error types.
- Relies on the authors' own design of counterfactual sampling; no ablation on the sampling procedure beyond the module ablations.
Experimental setup
Datasets: VQA-X A-OKVQA
Benchmarks: VQA-X A-OKVQA
Metrics: BLEU-4 METEOR ROUGE-L CIDEr SPICE answer accuracy human evaluation score
Baselines: PJ-X FME e-UG NLX-GPT S3C
Builds on: NLX-GPT CLIP GPT-2 Grad-CAM S3C
Closest papers by meaning
- Self-Critical Reasoning for Robust Visual Question Answering (NeurIPS 2019 · similarity 0.90)
- Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (CVPR 2017 · similarity 0.89)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.88)
- Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs (ICLR 2026 · similarity 0.88)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.88)
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.