Research map / Large language models / Language, safety and interpretability

BingoGuard: LLM Content Moderation Tools with Risk Levels

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces 11-topic severity rubrics and a generate-then-filter data pipeline to train BingoGuard, an LLM moderator that predicts both binary safety and 4-level severity, reporting state-of-the-art moderation results.

Problem

Existing LLM-based moderators make binary safe/unsafe judgments and struggle to assess risk severity, and training/evaluation data rarely annotate severity levels.

Approach

Define per-topic severity rubrics (levels 0-4) built from seven harm dimensions, generate responses at each severity level using specialized fine-tuned generators from small expert-audited seed sets, and iteratively refine training data with a moderator committee to mine hard examples.

Key results

• BingoGuard-llama3.1-8B reaches 79.4% overall detection accuracy on BingoGuardTest, above GPT-4o (76.5%) and ShieldGemma-7B (75.5%).
• On general benchmarks, BingoGuard-llama3.1-8B improves on the best baseline by 4.3% for query classification and 2.6% for response classification, with a 4.2% gain on WildGuardTest responses.
• Severity classification macro-F1 is 78.4 for BingoGuard-llama3.1-8B versus 54.3 for 5-shot GPT-4o.

Measured effects reported in the paper

Surprising findings

• Predictive probability of the 'unsafe' token is only weakly correlated with severity for LlamaGuard3, MD-Judge, and BingoGuard, and is over-confident (>0.9) for all levels.
• GPT-4o is not reliable as a filter for training data; the authors instead use a moderator committee for data refinement.

Limitations

• Model may miss unsafe content out of distribution from the defined policy and is not built for multilingual or dialectal content.
• Inter-annotator agreement on severity labels is moderate (Fleiss kappa 0.53), and severity guidelines may change with requirements.
• Only one round of iterative refinement was performed due to time and compute constraints.

Experimental setup

Datasets: WildGuardTrain BeaverTails SALAD-Bench SorryBench Do-not-answer DoAnythingNow WildChat LMSYS-Chat-1M HH-RLHF

Benchmarks: BingoGuardTest ToxicChat OpenAI Moderation AegisSafetyTest WildGuardTestPrompt XSTest BeaverTails Safe-RLHF WildGuardTestResponse HarmBench

Metrics: F1 detection accuracy macro-F1 (severity) Fleiss Kappa Krippendorff's alpha

Baselines: LlamaGuard2-8B LlamaGuard3-8B MD-Judge-7B WildGuard-7B ShieldGemma-7B GPT-4o Phi-3-mini

Builds on: WildGuard LlamaGuard3 Llama3.1-8B Phi-3-mini

Closest papers by meaning

• WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · similarity 0.89)
• From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring (NeurIPS 2025 · similarity 0.88)
• HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models (ICLR 2025 · similarity 0.87)
• SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.87)
• Look Before You Leap: Enhance Attention and Vigilance Regarding Harmful Content with GuidelineLLM (AAAI 2025 · similarity 0.87)
• Efficient Toxic Content Detection by Bootstrapping and Distilling Large Language Models (AAAI 2024 · similarity 0.87)