AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods

Ruixuan Huang, Xunguang Wang, Zongjie Li, Daoyuan Wu, Shuai Wang

ICLR 2026 · Large language models › Language, safety and interpretability · approach: Jailbreak attacks and defenses · 16 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use guideline-based GuidedEval instead of other LLM evaluatorsjailbreak evaluation · Invalid Rephrasing failuresFPR (%): HarmBench 63.57 -> GuidedEval 5.23Improved
use GuidedEval instead of StrongREJECTjailbreak evaluation · Misunderstanding failuresFPR (%): 38.82 -> 7.09Improved
use GuidedEval instead of other LLM-based systemsjailbreak evaluation · evaluator varianceaverage variance: 0.042932 -> 0.007701 vs StrongREJECTImproved
use GuidedEval guidelines to score responsesLLM judge · score distributionnormalized entropy: PAIR 0.25 -> GuidedEval 0.92Improved
use keyword-based evaluators instead of LLM-based onesjailbreak evaluation · agreement with LLM systemsagreement/rankingHurt
use PositiveKeyword instead of GuidedEvaljailbreak evaluation · Inconsistent Content failuresFPR (%): 5.64 -> 84.62Hurt
use GuidedEval versus prior evaluation on jailbreak methodsjailbreak ASR · GuidedBench vs prior benchmarksASR: prior >90% -> about 30%Hurt
use GuidedEval vs PAIR judge on General Advice failuresjailbreak evaluation · General Advice failuresFPR (%): 10.13 -> 9.07Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy