Research map / Vision-language and multimodal / Visual question answering
Bidirectional Contrastive Split Learning for Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bidirectional Contrastive Split Learning (BiCSL), a self-supervised split-learning framework that trains VQA models across decentralized clients via InfoNCE alignment between NHA and LTA adapter outputs, evaluated on VQA-v2 with five VQA architectures.
Problem
Privacy-preserving visual question answering when client data cannot be centralized and sharing full models exposes them to adversarial attacks; existing decentralized VQA (e.g., aimNet) relies on labeled answers and full model sharing.
Approach
Split a VQA model into client-side representation modules and a contrastive module; align a nonlinear head adapter (NHA) output on cross-modal features with a linear tail adapter (LTA) output on answer-text embeddings (GloVe-based APN) using InfoNCE, with gradient sharing across the split and averaged inter-client weight aggregation via a dual-server scheme.
Key results
- MMNas-l with BiCSL reaches 49.89% overall VQA-v2 validation accuracy vs 53.82% for centralized contrastive learning.
- Across seven model variants, a paired t-test is reported as showing no significant difference between BiCSL and centralized contrastive learning.
- BiCSL shows much stronger robustness to the dual-key Trojan attack than single fusion and split learning on MCAN-s (per the paper's figure, no numbers in text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BiCSL instead of centralized contrastive learning | VQA models on VQA-v2 · VQA-v2 validation, Overall | accuracy: 53.82% -> 49.89% | Hurt |
| Use BiCSL instead of centralized contrastive learning | BAN · VQA-v2 validation, Overall | accuracy: 36.23 $\pm$ 0.53 -> 35.11 $\pm$ 0.68 | Hurt |
| Use BiCSL instead of centralized contrastive learning | BUTD · VQA-v2 validation, Overall | accuracy: 45.08 $\pm$ 0.64 -> 40.96 $\pm$ 0.76 | Hurt |
| Use BiCSL instead of centralized contrastive learning | MFB · VQA-v2 validation, Overall | accuracy: 46.98 $\pm$ 0.58 -> 42.43 $\pm$ 0.72 | Hurt |
| Use BiCSL instead of centralized contrastive learning | MCAN-s · VQA-v2 validation, Overall | accuracy: 53.18 $\pm$ 0.61 -> 48.42 $\pm$ 0.68 | Hurt |
| Use BiCSL instead of centralized contrastive learning | MCAN-l · VQA-v2 validation, Overall | accuracy: 53.32 $\pm$ 0.55 -> 48.44 $\pm$ 0.62 | Hurt |
| Use BiCSL instead of centralized contrastive learning | MMNas-s · VQA-v2 validation, Overall | accuracy: 51.54 $\pm$ 0.57 -> 45.14 $\pm$ 0.69 | Hurt |
| Use BiCSL instead of centralized contrastive learning | MMNas-l · VQA-v2 validation, Overall | accuracy: 53.82 $\pm$ 0.53 -> 49.89 $\pm$ 0.61 | Hurt |
| Use BiCSL instead of centralized contrastive learning | VQA models on VQA-v2 · VQA-v2 validation under dual-key Trojan attack | robustness | Helped |
Surprising findings
- The paper's paired t-test is reported with a critical value of 2.477 and t=1.357 for a claimed non-significant difference, and the statistical framing is unusual.
Limitations
- Performance drop relative to centralized learning (e.g., 53.82% to 49.89% for MMNas-l).
- Robustness evaluated only against one dual-key Trojan attack; authors acknowledge more sophisticated attacks are future work.
- Experiments use only two clients with a random IID split.
Experimental setup
Datasets: VQA-v2 COCO
Benchmarks: VQA-v2
Metrics: ValAcc (overall, Yes/No, Number, Other accuracy) paired t-test attack accuracy under dual-key Trojan attack
Baselines: aimNet single fusion (centralized VQA) split learning QICE MMNas MCAN
Builds on: split learning federated learning InfoNCE MCAN MMNas
Closest papers by meaning
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.88)
- VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models (AAAI 2024 · similarity 0.88)
- Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA (AAAI 2024 · similarity 0.85)
- Towards Robust Visual Question Answering via Prompt-Driven Geometric Harmonization (AAAI 2025 · similarity 0.85)
- Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering (CVPR 2025 · similarity 0.85)
- SkillCLIP: Skill Aware Modality Fusion Visual Question Answering (Student Abstract) (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.