Research map / Large language models / Language, safety and interpretability
GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods
Ruixuan Huang, Xunguang Wang, Zongjie Li, Daoyuan Wu, Shuai Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use guideline-based GuidedEval instead of other LLM evaluators | jailbreak evaluation · Invalid Rephrasing failures | FPR (%): HarmBench 63.57 -> GuidedEval 5.23 | Improved |
| use GuidedEval instead of StrongREJECT | jailbreak evaluation · Misunderstanding failures | FPR (%): 38.82 -> 7.09 | Improved |
| use GuidedEval instead of other LLM-based systems | jailbreak evaluation · evaluator variance | average variance: 0.042932 -> 0.007701 vs StrongREJECT | Improved |
| use GuidedEval guidelines to score responses | LLM judge · score distribution | normalized entropy: PAIR 0.25 -> GuidedEval 0.92 | Improved |
| use keyword-based evaluators instead of LLM-based ones | jailbreak evaluation · agreement with LLM systems | agreement/ranking | Hurt |
| use PositiveKeyword instead of GuidedEval | jailbreak evaluation · Inconsistent Content failures | FPR (%): 5.64 -> 84.62 | Hurt |
| use GuidedEval versus prior evaluation on jailbreak methods | jailbreak ASR · GuidedBench vs prior benchmarks | ASR: prior >90% -> about 30% | Hurt |
| use GuidedEval vs PAIR judge on General Advice failures | jailbreak evaluation · General Advice failures | FPR (%): 10.13 -> 9.07 | Improved |
Closest papers by meaning
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.91)
- Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.90)
- Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models (AAAI 2025 · similarity 0.88)
- Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters (NeurIPS 2024 · similarity 0.88)
- A StrongREJECT for Empty Jailbreaks (NeurIPS 2024 · similarity 0.88)
- h4rm3l: A Language for Composable Jailbreak Attack Synthesis (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
