Research map / Multi-agent systems and games / LLM agents

There are Levels to It: Red Teaming LLMs with Hierarchical Reinforcement Learning

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formulates multi-turn LLM red teaming as a hierarchical MDP trained with PPO, using a high-level utterance-strategy policy and a low-level token policy with a subset-masking marginal-contribution token reward, attacking Llama-3.1-8B-Instruct and evaluating against several multi-turn and single-turn red teaming baselines.

Problem

Automated red teaming of LLMs is limited to templates or single-turn attacks, lacking sequential reward and long-horizon multi-turn attack planning, and suffers from sparse rewards and credit assignment challenges.

Approach

Frames red teaming as a hierarchical MDP: a high-level policy chooses an attack guide (template or LLM-generated) conditioned on full dialogue history, and a low-level token-level policy writes the utterance; token credit is assigned by marginal contribution via masking token subsets of size 1 and 2 measured with a guard model's harm score.

Key results

• On HarmBench with Llama-3.1-8B-Instruct under context-aware ASR@5, LLM-HL reaches 62.5% versus 41.0% for X-Teaming and 10.3% for WildTeaming.
• In the 30-step context-aware setting, LLM-HL achieves 95.0% HarmBench ASR on Llama-3.1-8B and 55.0% on GPT-4o, with mean TtS 4.1.
• Ablation shows full hierarchical training reaches 75.2 ASR@5 versus 31.2 for low-level only and 2.1 with no training.

Surprising findings

• Target harmfulness reward alone does not yield less-detectable attacks; the adversarial term -R(a) is needed.
• Baselines that perform well in myopic single-message settings degrade substantially when the target sees full conversation context.

Limitations

• Assumes a well-defined sequence-level harmfulness score, which may not hold for general or agentic tasks.
• Results depend on guard model strength; stronger guards during training improve the attacker.
• Publishing the work may make frontier LLMs more vulnerable to these attacks.

Experimental setup

Datasets: HarmBench

Benchmarks: HarmBench JailbreakBench WildBench

Metrics: ASR@5 ASR@30 turns to success perplexity diversity (pairwise cosine distance)

Baselines: Ferret WildTeaming HARM PAIR X-Teaming MTSA APRT GCG Rainbow Teaming

Builds on: PPO LoRA Crescendo Rainbow Teaming MART

Closest papers by meaning

• Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning (ICLR 2025 · similarity 0.92)
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (ICML 2024 · similarity 0.89)
• TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration (CVPR 2026 · similarity 0.89)
• Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.89)
• CoP: Agentic Red-teaming for Large Language Models using Composition of Principles (NeurIPS 2025 · similarity 0.88)
• DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints (AAAI 2025 · similarity 0.88)