Research map / Large language models / Language, safety and interpretability
MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Servers
Xuanjun Zong, Zhiqi Shen, Lei Wang, Yunshi Lan, Chao Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prepend a safety prompt to the user request | all models · MCP-SafetyBench | weighted ASR (%): 39.88 -> 38.65 | No clear effect |
| safety prompt | all models · Malicious Code Execution attacks | ASR: -21.54% | Improved |
| safety prompt | all models · Credential Theft attacks | ASR: -21.37% | Improved |
| safety prompt | all models · Function Overlapping attacks | ASR: +9.36% | Hurt |
| safety prompt | all models · Preference Manipulation attacks | ASR: +7.34% | Hurt |
| safety prompt | o4-mini · MCP-SafetyBench | ASR: -8.98% | Improved |
| safety prompt | open-source models (Kimi-K2) · MCP-SafetyBench | ASR: +5.30% | Hurt |
| add more MCP servers (2 to 4) (ablation) | GPT-4.1 · financial domain | TSR (%): 24.53 -> 30.19 | Improved |
| add more MCP servers (2 to 4) (ablation) | GPT-4.1 · financial domain | ASR (%): 43.40 -> 47.17 | Hurt |
| use reasoning vs non-reasoning model | LLM agents · MCP-SafetyBench | ASR | No clear effect |
| use open-source vs proprietary model | LLM agents · MCP-SafetyBench | ASR | No clear effect |
| higher task success rate | LLM agents · MCP-SafetyBench | defense success rate: r = -0.572 | Hurt |
Closest papers by meaning
- MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (NeurIPS 2026 · similarity 0.90)
- STAIR: Improving Safety Alignment with Introspective Reasoning (ICML 2025 · similarity 0.89)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.88)
- CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models (ICML 2025 · similarity 0.88)
- Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats (ICLR 2025 · similarity 0.88)
- UpSafe℃: Upcycling for Controllable Safety in Large Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
