Research map / Large language models / Language, safety and interpretability

On Calibration of LLM-based Guard Models for Reliable Content Moderation

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirically measures confidence calibration (ECE) of 9 open LLM-based guard models on 12 benchmarks for prompt and response classification, and evaluates post-hoc calibration (temperature scaling, contextual calibration, batch calibration).

Problem

LLM-based guard models for content moderation are evaluated mainly for classification accuracy, while their confidence calibration and reliability under jailbreaks and across response models are largely unexamined.

Approach

Systematic calibration audit of open-source guard models using ECE on binary prompt and response classification, followed by adapting post-hoc calibration methods (temperature scaling, contextual calibration, batch calibration) to guard models without requiring validation sets.

Key results

• WildGuard has the lowest average prompt-classification ECE of the untreated models at 14.4%, while MD-Judge has the lowest response-classification ECE at 11.4%; both remain above the 10% threshold considered poor.
• On Harmbench-adv prompt classification, WildGuard reaches F1 92.8% but ECE 34.9%.
• Contextual calibration lowers average prompt-classification ECE for Llama-Guard3 from 16.1% to 15.4%, and temperature scaling lowers average response-classification ECE for Llama-Guard3 from 24.0% to 22.0%.

Measured effects reported in the paper

Surprising findings

• A single 'unsafe' token as input can cause many guard models to confidently predict 'unsafe', while a space token often gets confident 'safe' predictions.
• Response-classification guard models often classify content-free space-token responses as unsafe when the user prompt is unsafe, indicating they act like prompt classifiers.

Limitations

• Only open-source guard models with accessible logits are studied; closed-source API moderation tools are excluded.
• Post-hoc methods only mitigate miscalibration in some scenarios; no single method works across all models and datasets.
• Temperature scaling is tuned on XSTest rather than in-domain validation data for most datasets.

Experimental setup

Datasets: OpenAI Moderation ToxicChat Aegis Safety Test SimpleSafetyTests XSTest HarmBench WildGuardMix BeaverTails SafeRLHF

Benchmarks: OpenAI Moderation ToxicChat Aegis Safety Test SimpleSafetyTests XSTest HarmBench Prompt WildGuardMix Test Prompt BeaverTails SafeRLHF HarmBench Response WildGuardMix Test Response Harmbench-adv

Metrics: ECE F1 FPR FNR

Baselines: Llama-Guard Llama-Guard2 Llama-Guard3 Aegis-Guard-Defensive Aegis-Guard-Permissive WildGuard HarmBench-Llama HarmBench-Mistral MD-Judge

Builds on: temperature scaling contextual calibration batch calibration

Closest papers by meaning

• On Evaluating the Durability of Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.90)
• WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · similarity 0.89)
• RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.89)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal (ICLR 2025 · similarity 0.89)
• Position: Building Guardrails for Large Language Models Requires Systematic Design (ICML 2024 · similarity 0.88)
• MoGU: A Framework for Enhancing Safety of LLMs While Preserving Their Usability (NeurIPS 2024 · similarity 0.88)