Research map / Large language models / Language, safety and interpretability
DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiveR-CT, a constrained-RL automatic red teaming method that relaxes the unsafe-score maximization into a threshold constraint and uses a dynamic k-NN semantic diversity reward to produce more diverse attacks at controllable ASR.
Problem
Automatic red teaming of LLM chat assistants optimizes for attack success rate, which sacrifices diversity of attack prompts and leads to reward overoptimization; existing history-based semantic rewards stagnate as history grows.
Approach
Cast red teaming as constrained policy optimization where the safety score is a threshold constraint (not a maximization target), solved with gradient descent ascent plus PPO with dynamic Lagrange multipliers, and replace the average-over-history semantic reward with a k-nearest-neighbor cosine reward (k=16) that shifts its target dynamically.
Key results
- At matched ASR of about 0.69, DiveR-CT reached Self-BLEU 0.834 and Vendi-Ngram 0.964 versus CRT's 0.802 and 0.559.
- DiveR-CT outperformed CRT on all lexical and semantic diversity metrics across three ASR levels (roughly 0.87, 0.69, 0.49).
- Against the test classifier, DiveR-CT maintained higher ASR than CRT and RL, indicating reduced overoptimization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use constrained safety objective with d_safe=-0.9 rather than maximizing unsafe score with CRT | DiveR-CT vs CRT · Red teaming, high ASR | Self-BLEU: 0.570 -> 0.746 | Helped |
| Use constrained safety objective with d_safe=-0.7 rather than CRT safety reward coefficient | DiveR-CT vs CRT · Red teaming, medium ASR | Vendi-Ngram: 0.559 -> 0.964 | Helped |
| Use constrained safety objective with d_safe=-0.5 rather than CRT safety reward coefficient | DiveR-CT vs CRT · Red teaming, low ASR | Vendi-Semantic: 0.083 -> 0.128 | Helped |
| Set safety threshold to d_safe=-0.9, -0.7, or -0.5 | DiveR-CT · Red teaming across ASR levels | ASR: 0.869; 0.686; 0.485 | Mixed |
| Constrain gibberish instead of maximizing gibberish reward (ablation) | DiveR-CT · Red teaming, d_safe=-0.7 | Self-BLEU: 0.811 -> 0.834 | Helped |
| Use top-16 semantic neighbors instead of all history neighbors (ablation) | DiveR-CT · Red teaming, d_safe=-0.7 | Vendi-Ngram: 0.896 -> 0.964 | Helped |
| Use top-16 semantic neighbors instead of all history neighbors (ablation) | DiveR-CT · Red teaming, d_safe=-0.7 | Semantic Mean: 0.411 -> 0.391 | Hurt |
| Use top-16 semantic neighbors instead of one neighbor (ablation) | DiveR-CT · Red teaming, d_safe=-0.7 | Vendi-Semantic: 0.113 -> 0.123 | Helped |
| Add the top-16 semantic-neighbor reward to CRT (ablation) | CRT · Red teaming, d_safe=-0.5 ASR group | Vendi-Ngram: 0.767 -> 0.848 | Helped |
| Add the top-16 semantic-neighbor reward to CRT (ablation) | CRT · Red teaming, d_safe=-0.5 ASR group | ASR: 0.444 -> 0.481 | Mixed |
| Safety-tune the blue team using DiveR-CT-generated red-team data rather than CRT- or RL-generated data | Blue-team safety tuning · AART, SAP, AdvenBench | safety robustness | Helped |
| Use DiveR-CT rather than CRT against a stronger blue-team model | Red-team generation · Llama-2-7b-chat-hf and Meta-Llama-3-8B-Instruct | diversity | Helped |
Surprising findings
- Constraint-satisfaction oscillations ("waving" Lagrange multipliers) were beneficial for diversity rather than harmful.
- Generation lengths decreased during training, contrary to typical RLHF patterns.
Limitations
- Single-turn interactions only; multi-turn vulnerabilities not studied.
- No domain knowledge incorporated; no fine-grained attack-category coverage.
- Results depend on classifier choice; safety-classifier changes altered the attack categories found.
Experimental setup
Datasets: tatsu-lab/alpaca PKU-Alignment/PKU-SafeRLHF
Benchmarks: Open LLM Leaderboard (Hellaswag, ARC-Challenge, TruthfulQA, Winogrande) AART SAP AdvBench PKU-SafeRLHF
Metrics: attack success rate Self-BLEU Vendi-Ngram semantic mean cosine distance Vendi-Semantic distinct-k entropy-k BLEU MS-Jaccard
Baselines: RL red teaming CRT Zero-shot sampling
Builds on: Curiosity-driven red teaming (CRT) PPO RL red teaming (Perez et al.)
Closest papers by meaning
- Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning (ICLR 2025 · similarity 0.92)
- TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration (CVPR 2026 · similarity 0.89)
- Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints (AAAI 2025 · similarity 0.89)
- Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents (ICML 2026 · similarity 0.88)
- Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.88)
- RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.