Research map / Large language models / AI and society
SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SafetyAnalyst, an interpretable, steerable LLM prompt safety moderation framework that generates CoT harm-benefit trees, aggregates them with 28 weights, and distills the tree generator into Llama-3.1-8B.
Problem
Current LLM safety moderation systems are black-box classifiers whose decisions are hard to explain and difficult to steer toward different safety preferences or community values.
Approach
Generate a structured harm-benefit tree (stakeholders, actions, effects with likelihood, extent, immediacy) via chain-of-thought, then aggregate its leaves with a transparent, 28-parameter weighted model whose weights can be top-down or bottom-up aligned to safety preferences; teacher-generated trees are distilled into two Llama-3.1-8B specialists.
Key results
- SafetyAnalyst achieves average weighted F1 of 81.2 across six prompt safety benchmarks, versus 71.7 for the next best (WildGuard) and 81.6 for zero-shot GPT-4.
- The distilled Llama-3.1-8B student performs comparably to teacher-generated data on WildJailbreak (F1 88.8 vs 87.7-91.8 for teachers).
- Average inference time is 6.12 seconds per prompt on one H100, versus 0.22 seconds for WildGuard.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add harm features to the aggregation (ablation) | SafetyAnalyst · WildGuardTest vanilla, WildGuardTest adversarial, and WildJailbreak vanilla | F1: 78.6 vs 90.9; 71.1 vs 79.6; 76.6 vs 88.8 | Helped |
| Add benefit features to the aggregation (ablation) | SafetyAnalyst · WildGuardTest vanilla, WildGuardTest adversarial, and WildJailbreak vanilla | F1: 89.9 vs 90.9; 79.6 vs 79.6; 88.5 vs 88.8 | Mixed |
| Add action features to the aggregation (ablation) | SafetyAnalyst · WildGuardTest vanilla, WildGuardTest adversarial, and WildJailbreak vanilla | F1: 88.6 vs 90.9; 78.8 vs 79.6; 87.4 vs 88.8 | Helped |
| Add effect features to the aggregation (ablation) | SafetyAnalyst · WildGuardTest vanilla, WildGuardTest adversarial, and WildJailbreak vanilla | F1: 79.3 vs 90.9; 71.6 vs 79.6; 79.2 vs 88.8 | Helped |
| Add extent features to the aggregation (ablation) | SafetyAnalyst · WildGuardTest vanilla, WildGuardTest adversarial, and WildJailbreak vanilla | F1: 90.7 vs 90.9; 80.0 vs 79.6; 89.1 vs 88.8 | Mixed |
| Add likelihood features to the aggregation (ablation) | SafetyAnalyst · WildGuardTest vanilla, WildGuardTest adversarial, and WildJailbreak vanilla | F1: 82.0 vs 90.9; 74.4 vs 79.6; 76.6 vs 88.8 | Helped |
| Add immediacy features to the aggregation (ablation) | SafetyAnalyst · WildGuardTest vanilla, WildGuardTest adversarial, and WildJailbreak vanilla | F1: 90.5 vs 90.9; 80.5 vs 79.6; 89.3 vs 88.8 | Mixed |
| Use the SafetyAnalyst student model rather than GPT-4o-generated features | Aggregation model on WildJailbreak · WildJailbreak held-out test set | F1: 91.8 -> 88.8 | Hurt |
| Distill harm-benefit tree generation into the student model | SafetyAnalyst · WildGuardTest and public prompt safety benchmarks | F1 | No clear effect |
Surprising findings
- Beneficial effects receive only about 7.59% of the relative weight of harmful effects in the aggregation aligned to WildJailbreak, suggesting benefits are underrepresented in ground-truth labels.
- Ablating harm features drops WildGuardTest adversarial F1 from 79.6 to 71.1, while ablating likelihood drops WildJailbreak F1 from 88.8 to 76.6.
Limitations
- Harm-benefit tree generation causes substantially longer inference time than label-only moderation systems.
- Simply summing harmful and beneficial effects may be insufficient for safe decisions; incommensurability of values is not addressed.
- Pluralistic/steerable alignment is proposed but not validated on diverse human populations; annotator agreement with generated features varies (about 54-77%).
Experimental setup
Datasets: WildJailbreak WildChat AegisSafetyTrain SimpleSafetyTests HarmBench WildGuardTest AIR-Bench-2024 SORRY-Bench
Benchmarks: SimpleSafetyTests HarmBench WildGuardTest AIR-Bench-2024 SORRY-Bench
Metrics: F1 AUPRC AUROC
Baselines: OpenAI Moderation API Llama-Guard Llama-Guard-2 Llama-Guard-3 Aegis-Guard-Defensive Aegis-Guard-Permissive ShieldGemma-2B ShieldGemma-9B ShieldGemma-27B WildGuard GPT-4
Builds on: WildJailbreak AIR 2024 risk taxonomy QLoRA symbolic knowledge distillation
Closest papers by meaning
- Safety Pretraining: Toward the Next Generation of Safe AI (NeurIPS 2025 · similarity 0.87)
- Position: Require Frontier AI Labs To Release Small "Analog" Models (NeurIPS 2025 · similarity 0.86)
- Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? (NeurIPS 2024 · similarity 0.86)
- Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention (ICLR 2026 · similarity 0.85)
- SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment (NeurIPS 2025 · similarity 0.85)
- On the Role of Attention Heads in Large Language Model Safety (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.