Research map / Large language models / Language, safety and interpretability

MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Jailbreak attacks and defenses · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MT-JailBench, a modular framework that decomposes multi-turn jailbreaks into five components and evaluates existing attacks under controlled budgets and judges, then recomposes top components into CrescendoX.

Problem

Multi-turn jailbreak attacks are usually evaluated as black-box pipelines with different budgets, judges, retry rules, and strategy generation, so it is unclear which components drive reported attack success rate.

Approach

Represent each multi-turn attack as five interchangeable modules (evaluation function, attack strategy, prompt generation, prompt refinement, flow control) under shared resource budgets and judges, enabling component-level ablation and recomposition into a stronger attack (CrescendoX).

Key results

• Under limited budgets (max 10 interactions), ActorBreaker drops by up to 35.9 points (Llama-3-70B) and CoA by 35.2 points (GPT-5), while Crescendo and FITD are comparatively stable.
• Requiring agreement of all three judges lowers FITD ASR on GPT-4o from 86.8% to 17.6%, showing judge choice strongly changes measured success.
• CrescendoX (Crescendo prompt generation + XTeaming refinement and flow control) achieves the highest ASR on 19 of 21 target models, exceeding Crescendo by over 20 points on several targets.

Surprising findings

• Judge choice can change measured FITD ASR on GPT-4o from 86.8% to 17.6% under cross-validation.
• Stochastic sampling from a single fixed strategy (Crescendo) can match explicit strategy enumeration (XTeaming) in eventual success.

Limitations

• Evaluation limited to a black-box, text-only threat model and HarmBench behaviors (159).
• Relies on LLM-based judges whose boundaries may differ from ground truth.
• Absolute ASRs reflect system-level deployment safeguards and are not absolute measures of model safety.

Experimental setup

Datasets: HarmBench

Benchmarks: HarmBench

Metrics: attack success rate

Baselines: Crescendo CoA FITD ActorBreaker XTeaming

Builds on: Crescendo Chain-of-Attack Foot-in-the-Door ActorBreaker X-Teaming

Closest papers by meaning

• Systematic Scaling Analysis of Jailbreak Attacks in Large Language Models (NeurIPS 2026 · similarity 0.93)
• JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.91)
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.91)
• Multi-Turn Jailbreaking Large Language Models via Attention Shifting (AAAI 2025 · similarity 0.91)
• Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.91)
• Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.90)