Research map / Vision-language and multimodal / Multimodal LLMs
TrustGen: A Platform of Dynamic Benchmarking on the Trustworthiness of Generative Foundation Models
Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang, Jiayi Ye, Yujun Zhou, Yanbo Wang, Jiawen Shi, Qihui Zhang, Han Bao, Zhaoyi Liu and 52 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TrustGen, a platform for dynamic benchmarking of the trustworthiness of generative foundation models, addressing fragmented and outdated existing evaluations.
Problem
Assessing the trustworthiness of generative foundation models (LLMs and text-to-image systems) is critical but existing evaluation efforts are fragmented, quickly outdated, and lack extensibility.
Approach
A dynamic benchmarking platform for trustworthiness of GenFMs intended to stay current and extensible, rather than a static fixed benchmark suite.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use open-source CogView-3-Plus instead of proprietary DALL-E-3 | text-to-image model · TrustGen T2I average | trustworthiness score: 72.52 -> 74.96 | Improved |
| use proprietary DALL-E-3 instead of open-source T2I models | text-to-image model · T2I Truthfulness | trustworthiness score: 44.80 vs 28.06-35.67 for open models | Improved |
| use proprietary vs open-source LLM (Llama-3.2-70B vs GPT-4o) | LLM · TrustGen LLM trustworthiness | trustworthiness score | No clear effect |
| scale up VLM size (Llama-3.2-11B-V to 90B-V) | VLM · TrustGen VLM average | trustworthiness score: 64.91 -> 47.33 | Hurt |
| larger LLM size within a series | LLM · LLM privacy | privacy preservation | Hurt |
| larger LLM size within a series | LLM · LLM fairness | fairness metrics | Improved |
| larger VLM size | VLM · VLM privacy | privacy score | Mixed |
| apply input perturbations | LLMs and VLMs · robustness | performance | Mixed |
| evaluate on dynamically generated vs established benchmark data | LLMs · truthfulness | performance | Improved |
| use proprietary instead of open-source VLMs | VLM · jailbreak safety | refusal-to-answer rate | Improved |
Closest papers by meaning
- MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models (NeurIPS 2024 · similarity 0.91)
- VMDT: Decoding the Trustworthiness of Video Foundation Models (NeurIPS 2025 · similarity 0.89)
- More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness (ICLR 2025 · similarity 0.89)
- Position: TrustLLM: Trustworthiness in Large Language Models (ICML 2024 · similarity 0.89)
- Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression (ICML 2024 · similarity 0.87)
- CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
