Research map / Vision-language and multimodal / Human and embodied modeling
Adventures of Trustworthy Vision-Language Models: A Survey
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A survey of vision-language transformers (VLPMs) through the lens of trustworthiness, covering bias/fairness, robustness, and interpretability/explainability, and listing open challenges.
Problem
Vision-language pre-trained models and vision transformers are widely deployed, but their trustworthiness (bias, robustness, interpretability) has been studied mainly for unimodal models or CNNs, with few collective reviews.
Approach
Organizes existing studies of vision-language transformers under three responsible-AI principles (bias, robustness, interpretability), summarizes metrics and methods for each, and identifies gaps such as architecture-specific attacks, standard protocols, attention reliability, and language-dominance of VLPMs.
Key results
- Reports that published vision-language transformer research on vision transformers grew roughly 15x (vision transformer) and 9.5x (vision-language transformer) from 2018 to 2022 based on keyword analysis.
- Summarizes that bias studies such as VLStereoSet find state-of-the-art VLPMs encode stereotypical bias across gender, profession, race, and religion.
- Concludes that VLPMs rely more on language than vision, and that attention is not reliably an explanation, based on cited studies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tokens for learning high-frequency patterns | Vision Transformers · Image classification | accuracy | Helped |
| Add tokens for learning high-frequency patterns | Vision Transformers · Adversarial settings | robustness | Hurt |
| Use sufficient training data | Vision Transformers compared with ResNets · Image classification | performance | Helped |
| Use different fusion strategies for missing or incomplete modalities | VLPMs · Missing-modality settings | accuracy | Helped |
| Use different multimodal fusion strategies | VLPMs · Missing-modality settings | robustness | Mixed |
| Apply text-guided contrastive adversarial training (TeCoA) with finetuning | Vision-language models · Zero-shot adversarial settings | robustness | Helped |
Surprising findings
- Cited studies find transformers are not more robust than CNNs in some settings, contrary to prevailing claims, and attacks designed for CNNs may be unfairly applied to transformers.
Limitations
- Relies on cited works and does not run new experiments or a unified benchmark comparison
- Acknowledges lack of standard metrics and common protocols across bias and robustness studies
Closest papers by meaning
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.87)
- TRUST-VLM: Thorough Red-Teaming for Uncovering Safety Threats in Vision-Language Models (ICML 2025 · similarity 0.86)
- ProTransformer: Robustify Transformers via Plug-and-Play Paradigm (NeurIPS 2024 · similarity 0.86)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.85)
- Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models (ICLR 2025 · similarity 0.85)
- Improving Interpretation Faithfulness for Vision Transformers (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.