Research map / Large language models / Instruction and fine-tuning

Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GFlowNet fine-tuning followed by MLE smoothing on collected high-reward prompts to train a GPT-2 attacker that generates diverse and effective red-teaming prompts against several target LLMs.

Problem

Automated red-teaming with RL tends to mode-collapse or produce ineffective attacks, so it fails to find diverse, effective, and transferable attack prompts against LLMs, especially safety-tuned ones.

Approach

Treat attack generation as sampling from a tempered reward distribution using GFlowNet trajectory balance fine-tuning, then smooth by re-fine-tuning the pretrained attacker with MLE on high-reward prompts collected during GFlowNet training, which also allows cheap reranking-based adaptation to new targets.

Key results

• GFlowNet + MLE is the only method that balances high toxicity rate with high diversity across the four main target LLMs in the reported experiments.
• On Gemma-2b-it, GFlowNet + MLE attains 85.16% toxicity rate, versus 0.00% for PPO + Novelty and 98.45% for REINFORCE (which collapses to near-identical prompts and transfers poorly).
• Prompts generated against Gemma-2b-it transfer to other LLMs at 22.94% to 77.16% toxicity, e.g. 77.16% on Gemma-1.1-2b-it, outperforming baselines on most targets except Mistral-7b-instruct-v0.2; 65% of prompts from a Llama-2-7b-chat attacker elicited harmful responses from GPT-4o.

Measured effects reported in the paper

Surprising findings

• PPO + Novelty generated no prompts with toxicity above 0.2 against Gemma-2b-it and Llama-2-7b-chat, and produced no high-reward samples to build an MLE dataset.
• With a RoBERTa hate-speech classifier as reward, the attacker exploits reward hacking, producing prompts that are flagged toxic but do not elicit toxic responses under Llama-Guard.

Limitations

• Performance is bounded by the toxicity classifier, which is a proxy for true harm that is subjective and context-dependent.
• Online RL training requires sampling multiple target responses per prompt, which is costly.
• Evaluated on toxicity only; the jailbreaking setting and multimodal red-teaming are left to future work.

Experimental setup

Datasets: SafetyDataset AdvBench Open LLM Leaderboard tasks (ARC, HellaSwag, TruthfulQA, MMLU, Winogrande, GSM8k)

Benchmarks: Red-teaming toxicity rate and diversity on GPT-2, Dolly-v2-7b, Gemma-2b-it, Llama-2-7b-chat, Llama-3.1-8B-Instruct Transfer toxicity to Llama-2/3, Gemma, Mistral, Starling, GPT-4o Open LLM Leaderboard (ARC, HellaSwag, TruthfulQA, MMLU, Winogrande, GSM8k)

Metrics: toxicity rate average pairwise cosine distance transfer toxicity rate

Baselines: SFT ICL REINFORCE PPO + Novelty GFlowNet

Builds on: GFlowNet trajectory balance REINFORCE PPO Rainbow Teaming

Closest papers by meaning

• DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints (AAAI 2025 · similarity 0.92)
• There are Levels to It: Red Teaming LLMs with Hierarchical Reinforcement Learning (NeurIPS 2026 · similarity 0.92)
• Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints (AAAI 2025 · similarity 0.91)
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (ICML 2024 · similarity 0.90)
• Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.89)
• TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration (CVPR 2026 · similarity 0.89)