Research map / Large language models / Language, safety and interpretability
RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RigorLLM, a guardrail for LLM input/output moderation that fuses probabilistic KNN over energy-based (Langevin) generated embeddings with a LlamaGuard-based LLM classifier, plus a minimax-optimized safe suffix and prompt augmentation.
Problem
LLM content moderation guardrails (OpenAI moderation, Perspective, NeMo Guardrails, LlamaGuard) generalize poorly to out-of-distribution harmful content and are vulnerable to jailbreak adversarial suffixes.
Approach
Combine a KNN classifier over embeddings augmented with energy-based Langevin-dynamics generated soft samples (constrained by similarity and fluency) with a fine-tuned LLM guard, add an optimized safe suffix (minimax GCG) and LLM-generated paraphrase/summary augmentations, and aggregate probabilities by weighted average.
Key results
- On ToxicChat, RigorLLM reaches AUPRC 0.869 and F1 0.749 vs. 0.798 / 0.609 for LlamaGuard (best baseline).
- On AdvBench under GCG (U1), GCG (V), and AutoDAN attacks, RigorLLM achieves HDR of 1.00 while LlamaGuard drops to 0.65-0.79.
- On OpenAI Moderation, RigorLLM reaches AUPRC 0.841 and F1 0.791, above OpenAI API (0.836 / 0.765).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RigorLLM instead of the strongest baseline | harmful-content moderation · ToxicChat | AUPRC: 0.798 -> 0.869 | Helped |
| Use RigorLLM instead of the strongest baseline | harmful-content moderation · ToxicChat | F1: 0.609 -> 0.749 | Helped |
| Use RigorLLM instead of the strongest baseline | harmful-content moderation · OpenAI Moderation Dataset | F1: 0.765 -> 0.791 | Helped |
| Use RigorLLM instead of LlamaGuard | moderation under adversarial strings · AdvBench Harmful Behavior | HDR | Helped |
| Add LlamaGuard aggregation (ablation) | RigorLLM · OpenAI Moderation Dataset | F1: 0.731 -> 0.791 | Helped |
| Add KNN aggregation (ablation) | RigorLLM · AdvBench Harmful Behavior, GCG (U1) | HDR: 0.81 -> 1.00 | Helped |
| Add prompt augmentation (ablation) | RigorLLM · OpenAI Moderation Dataset | F1: 0.723 -> 0.791 | Helped |
| Add safe suffix (ablation) | RigorLLM · OpenAI Moderation Dataset | F1: 0.784 -> 0.791 | Helped |
| Add safe suffix (ablation) | RigorLLM · AdvBench Harmful Behavior, GCG (U1) | HDR: 0.96 -> 1.00 | Helped |
| Remove LlamaGuard aggregation (ablation) | RigorLLM · AdvBench Harmful Behavior, GCG (U1) | HDR: 1.00 -> 1.00 | No clear effect |
| Remove prompt augmentation (ablation) | RigorLLM · AdvBench Harmful Behavior, GCG (U1) | HDR: 1.00 -> 1.00 | No clear effect |
| Increase the number of generated training examples (ablation) | RigorLLM energy-based data generation · content moderation | AUPRC | Helped |
Limitations
- Adaptive attacks specifically targeting RigorLLM were not evaluated; the authors note future strong adaptive attacks could break it.
- Per-category comparison with OpenAI API is hard due to taxonomy mismatch.
- Hyperparameters k and alpha selected by grid search, and the adversarial evaluation uses only 100 AdvBench behaviors with a few fixed suffixes.
Experimental setup
Datasets: HEx-PHI HotpotQA MT-bench OpenAI Moderation Dataset ToxicChat AdvBench
Benchmarks: OpenAI Moderation Dataset ToxicChat AdvBench Harmful Behaviors with GCG and AutoDAN attacks
Metrics: AUPRC F1 HDR (harmful content detection rate)
Baselines: OpenAI API Perspective API NeMo Guardrails LlamaGuard Vicuna-7B
Builds on: GCG LlamaGuard energy-based models NeMo Guardrails
Closest papers by meaning
- BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.91)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.89)
- Look Before You Leap: Enhance Attention and Vigilance Regarding Harmful Content with GuidelineLLM (AAAI 2025 · similarity 0.89)
- Protecting Your LLMs with Information Bottleneck (NeurIPS 2024 · similarity 0.89)
- Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.88)
- Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.