Research map / Large language models / Language, safety and interpretability
Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DART, a text-diffusion-inspired black-box red-teaming method that perturbs reference prompts in embedding space under a norm budget, yielding harmful test prompts close to the reference.
Problem
Automated red-teaming of LLMs is hard to control; red-teamers need to find harmful inputs that stay in close proximity to a given reference prompt dataset (topics, styles, harm types).
Approach
Train a T5-based diffusion model to output a noise vector added to the reference prompt's sentence embedding, with an L2-norm regularizer enforcing the proximity budget; the perturbed embedding is decoded to text with vec2text and trained with PPO against a toxicity classifier reward.
Key results
- DART with epsilon=0.1 on Vicuna-7b reaches 2.49% ASR with 0.86 cosine similarity versus 1.58% unmodified, while RL baseline at comparable similarity (alpha=0.25) gets 0.85% ASR.
- On GPT2-alpaca, DART (epsilon=0.5) achieves 15.38% ASR at cosine similarity 0.73, versus 1.83% for Zero-Shot at 0.69 similarity.
- Llama-2-7b-chat is hard to attack within proximity: best close-proximity DART ASR is about 1.07% at cosine 0.73.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DART rather than the proximity-constrained RL baseline | red-teaming prompts across GPT2-alpaca, Vicuna-7b, and Llama-2-7b · Red Teaming dataset and Alpaca dataset | ASR | Helped |
| Increase DART's proximity budget from ε=0.1 to ε=0.5 or ε=2 (ablation) | DART · Red Teaming dataset and Alpaca dataset | ASR | Helped |
| Remove the proximity penalty from the RL reward and use KL-divergence regularization only (ablation) | RL red-teaming baseline · Vicuna-7b | Reward: -8.63 -> -8.60 | No clear effect |
| Remove the proximity penalty from the RL reward (ablation) | RL red-teaming baseline · Vicuna-7b | Cosine similarity: 0.15 | Hurt |
| Use Few-Shot rather than Zero-Shot prompting | uncensored Llama2-7b-chat-hf red-teaming model · Red Teaming dataset and Alpaca dataset | ASR | Mixed |
Surprising findings
- Removing the cosine-similarity penalty in the RL baseline did not change reward but made the proximity budget uncontrollable (cosine 0.15 vs budget 0.5).
- Zero- and few-shot and FLIRT baselines are highly volatile and regularly exceed the proximity budget.
Limitations
- Most discovered prompts contain small errors such as typos or grammatical mistakes.
- Requires manual selection of the budget hyperparameter epsilon.
- Only single-turn conversations were evaluated.
Experimental setup
Datasets: Red Teaming dataset (Anthropic HH red-team) alpaca-gpt4 Beavertails PAWS
Benchmarks: Red Teaming dataset alpaca-gpt4 prompts Beavertails
Metrics: mean toxicity reward (classifier logits) attack success rate (toxicity prob > 50%) cosine similarity intent maintained (manual annotation)
Baselines: Unmodified RL (auto-regressive PPO) Zero-Shot Few-Shot FLIRT
Builds on: proximal policy optimization vec2text continuous text diffusion FLIRT
Closest papers by meaning
- Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning (ICLR 2025 · similarity 0.91)
- Curiosity-driven Red-teaming for Large Language Models (ICLR 2024 · similarity 0.89)
- DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints (AAAI 2025 · similarity 0.89)
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (ICML 2024 · similarity 0.89)
- RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates (ICML 2026 · similarity 0.87)
- Red-Teaming Text-to-Image Systems by Rule-based Preference Modeling (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.