Research map / Vision-language and multimodal / Multimodal LLMs
SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models
Zonghao Ying, Aishan Liu, Siyuan Liang, Lei Huang, Jieyi Guo, Wenbo Zhou, Xianglong Liu, Dacheng Tao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add collaborative deliberation step to jury evaluation (ablation) | Jury Deliberation Protocol · MiniBench | Cohen's Kappa: 0.37 -> 0.61 | Improved |
| use JDP rather than other evaluators for agreement with human annotation | safety evaluator · MiniBench | Cohen's Kappa: 0.89 | Improved |
| scale model from 2B to 7B | Qwen2-VL · MiniBench | ASR (%): 43.1 -> 25.8 | Improved |
| scale model from 7B to 13B | LLaVA-v1.6-Vicuna · MiniBench | ASR (%): 31.7 -> 15.0 | Improved |
| use GPT-4o instead of GPT-4o-mini | GPT family · SafeBench | SRI: 98.4 -> 96.1 | Hurt |
| apply LPT jailbreak to MiniBench queries | Bunny-v1.1 · MiniBench | ASR (%): 35.2 -> 70.0 | Hurt |
| apply LPT jailbreak to MiniBench queries | Qwen-VL-Chat · MiniBench | ASR (%): 10.4 -> 68.0 | Hurt |
| replace black/white/noise/random images with semantically related MiniBench images (ablation) | MLLM safety evaluation · MiniBench | ASR / SRI | Improved |
| wrap the LM backbone into a multimodal LLM (ablation) | MLLMs vs backbone LMs · MiniBench | ASR (%) | Hurt |
| use audio plus text input instead of audio only | Qwen2-Audio · SafeBench audio (male voice) | ASR (%): 12.0 -> 58.2 | Hurt |
Closest papers by meaning
- MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.95)
- SDEval: Safety Dynamic Evaluation for Multimodal Large Language Models (AAAI 2026 · similarity 0.93)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.92)
- CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models (ICML 2025 · similarity 0.91)
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.90)
- Multimodal Situational Safety (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
