Research map / Large language models / Language, safety and interpretability

Automated Red Teaming with GOAT: the Generative Offensive Agent Tester

ICML 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GOAT, an attacker LLM that reasons (observation, thought, strategy) and layers seven prompt-level jailbreak techniques over multi-turn conversations with target LLMs, outperforming Crescendo on JailbreakBench.

Problem

Automated jailbreaking research mostly seeks single adversarial prompts, while real users exploit multi-turn conversations; existing multi-turn automated attacks (e.g., Crescendo) are limited in combining diverse human-style techniques.

Approach

A general-purpose helpful-only attacker LLM is given natural-language definitions of prompt-level attacks in its system prompt (in-context, no fine-tuning) and uses chain-of-attack-thought reasoning (observation, thought, strategy, response) to adaptively chain attacks across turns against a closed-box target.

Key results

• GOAT achieves ASR@10 of 97% against Llama 3.1 8B and 88% against GPT-4-Turbo on JailbreakBench within 5 conversation turns.
• GOAT outperforms Crescendo on ASR@1 and ASR@10 for every target model, but performs worse on Llama 2 7B, reaching 85% ASR@10 there.
• GOAT uses no separate judge during conversation, making roughly 3 times fewer LLM calls per turn than Crescendo as stated by the authors.

Measured effects reported in the paper

Surprising findings

• The attacker model is a general helpful-only model with no explicit red-teaming training, relying solely on in-context attack definitions.

Limitations

• Five-turn cap set by the smallest target's 4096-token context, which may limit attacks that need more turns to recover.
• Attack success depends heavily on the attacker model's reasoning quality and lack of safety tuning.
• Attack definitions are only partially released (names and high-level definitions), and the judge is a single Llama 3.0 70B evaluator with debated violation criteria.

Experimental setup

Datasets: JailbreakBench

Benchmarks: JailbreakBench

Metrics: ASR@1 ASR@10

Baselines: Crescendo

Builds on: Crescendo Chain-of-Thought prompting HarmBench AdvBench

Closest papers by meaning

• TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration (CVPR 2026 · similarity 0.89)
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (ICML 2024 · similarity 0.89)
• Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts (NeurIPS 2024 · similarity 0.89)
• AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration (NeurIPS 2025 · similarity 0.89)
• Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning (ICLR 2025 · similarity 0.89)
• There are Levels to It: Red Teaming LLMs with Hierarchical Reinforcement Learning (NeurIPS 2026 · similarity 0.89)