Research map / Large language models / Language, safety and interpretability
Toxicity Detection for Free
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MULI, a free toxicity detector for prompts that applies sparse logistic regression to the first response token's logits of the target LLM itself, outperforming LlamaGuard and the OpenAI Moderation API at low FPR.
Problem
Toxicity detectors for LLM prompts are costly (extra model call per query) and have low TPR at very low FPR under real-world class imbalance, while aligned LLMs' refusal signals go unused.
Approach
Toxic prompts raise the probability of refusal-starting tokens (e.g., "Sorry", "Cannot", "I") in the LLM's first response-token distribution, even when the model still answers; a lasso-regularized logistic regression over the full first-token log-odds vector turns this into a near-zero-cost detector.
Key results
- On ToxicChat, MULI reaches 42.54% TPR at 0.1% FPR versus 5.25% for LlamaGuard and 6.08% for OpenAI Moderation API, with AUPRC 91.29 vs 70.14 for LlamaGuard.
- On LMSYS-Chat-1M, MULI achieves 66.85% TPR at 0.1% FPR versus 7.29% for LlamaGuard, and is comparable to the OpenAI Moderation API overall.
- Training on only ten prompts (one toxic) still gives 76.92 AUPRC and 13.81% TPR at 0.1% FPR on ToxicChat, beating LlamaGuard and OMod on TPR at that FPR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MULI instead of LlamaGuard | toxicity detection · ToxicChat test | TPR@FPR 0.1%: 5.25 -> 42.54 | Helped |
| use MULI instead of OMod | toxicity detection · ToxicChat test | AUPRC: 63.14 -> 91.29 | Helped |
| use MULI instead of LlamaGuard | toxicity detection · LMSYS-Chat-1M test | TPR@FPR 0.1%: 7.29 -> 66.85 | Helped |
| use MULI instead of OMod | toxicity detection · LMSYS-Chat-1M test | TPR@FPR 0.01%: 56.95 -> 53.62 | Hurt |
| use MULI instead of LlamaGuard | toxicity detection · OpenAI Moderation API Evaluation | TPR@FPR 0.1%: 14.56 -> 24.90; 14.56 -> 25.86 | Helped |
| use raw logits instead of f* (ablation) | MULI · ToxicChat test | TPR@FPR 0.1%: 42.54 -> 45.03 | Helped |
| use probabilities instead of f* (ablation) | MULI · ToxicChat test | AUPRC: 91.29 -> 36.50 | Hurt |
| use log probabilities instead of f* (ablation) | MULI · ToxicChat test | AUPRC: 91.29 -> 91.28 | No clear effect |
| replace L1 regularization with L2 regularization (ablation) | MULI with f* · ToxicChat test | AUPRC: 91.29 -> 90.66 | Hurt |
| remove sparse regularization (ablation) | MULI with f* · ToxicChat test | AUPRC: 91.29 -> 89.05 | Hurt |
| increase the training set from 10 examples to 5082 | MULI · ToxicChat | AUPRC: 76.92 -> 91.29 | Helped |
| use Llama2-13B instead of Llama2-7B as the base model | MULI · ToxicChat | TPR@FPR 0.1%: 43.92 -> 46.13 | Helped |
Surprising findings
- Even training on just ten examples (one toxic) yields a detector that beats LlamaGuard and OpenAI Moderation API on TPR at 0.1% FPR on ToxicChat.
- Detector performance correlates with the base LLM's security score against jailbreaks (1 - HarmBench ASR), so better-aligned models give better detectors.
Limitations
- Relies on the base LLM being well aligned; performance drops with weakly aligned or unaligned models.
- Not tested against malicious fine-tuning to remove safety alignment or adversarial attacks; a separate LLM detector may be needed there.
- Not evaluated for fairness across demographic subgroups.
Experimental setup
Datasets: ToxicChat LMSYS-Chat-1M OpenAI Moderation API Evaluation dataset HarmBench
Benchmarks: ToxicChat LMSYS-Chat-1M OpenAI Moderation API Evaluation dataset
Metrics: Acc_opt AUPRC TPR@FPR 10% TPR@FPR 1% TPR@FPR 0.1% TPR@FPR 0.01%
Baselines: LlamaGuard OpenAI Moderation API GPT-4o GPT-4o-mini Logits_Cannot
Builds on: Llama Guard OpenAI Moderation API
Closest papers by meaning
- WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · similarity 0.87)
- Efficient Toxic Content Detection by Bootstrapping and Distilling Large Language Models (AAAI 2024 · similarity 0.85)
- From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring (NeurIPS 2025 · similarity 0.85)
- BingoGuard: LLM Content Moderation Tools with Risk Levels (ICLR 2025 · similarity 0.84)
- LLMScan: Causal Scan for LLM Misbehavior Detection (ICML 2025 · similarity 0.84)
- SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.