Research map / Large language models / Language, safety and interpretability
RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RSafe, a reasoning-based LLM guard that uses policy-guided step-by-step reasoning plus zero-shot GRPO with rule-based format and accuracy rewards, trained on about 10K public examples to match SOTA guards and generalize better OOD.
Problem
Guard models for LLM input/output moderation rely on large human-curated datasets and generalize poorly to emerging harmful categories and jailbreak attacks.
Approach
Elicit policy-conditioned step-by-step safety reasoning from a backbone LLM via an instruction prompt that accepts user-specified safety taxonomies, then align it with zero-shot rule-based RL (GRPO) using only format and outcome accuracy rewards, with no SFT cold start.
Key results
- RSafe reaches weighted average accuracy 0.904 and F1 0.889 across six moderation benchmarks, versus 0.899 / 0.889 for GuardReasoner-8B.
- On WildGuardTest adversarial subset, RSafe scores F1 0.668 and RSafe-adaptive F1 0.717, above ShieldGemma-9B-adaptive (0.711) and LlamaGuard-8B (0.609) in F1 for the adversarial split of RSafe-adaptive only.
- On MedSafetyBench, RSafe-adaptive accuracy rises to 0.804 from RSafe's 0.588, approaching GPT-4o-as-judge at 0.881.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reinforced alignment (RL) to RSafe (ablation) | RSafe · Weighted Avg. | accuracy: 0.830 -> 0.904 | Helped |
| Add reinforced alignment (RL) to RSafe (ablation) | RSafe · Weighted Avg. | F1: 0.807 -> 0.889 | Helped |
| Add guided reasoning to RSafe (ablation) | RSafe · Weighted Avg. | accuracy: 0.846 -> 0.904 | Helped |
| Add guided reasoning to RSafe (ablation) | RSafe · Weighted Avg. | F1: 0.813 -> 0.889 | Helped |
| Provide the ground-truth safety taxonomy at inference (ablation) | RSafe · WildGuardTest overall | F1: 0.772 -> 0.823 | Helped |
| Provide the ground-truth safety taxonomy at inference (ablation) | RSafe · WildGuardTest adversarial | F1: 0.668 -> 0.717 | Helped |
| Use RSafe rather than GuardReasoner-8B | Safety moderation · Aegis | accuracy: 0.812 -> 0.834 | Helped |
| Use RSafe rather than WildGuard-7B | Safety moderation · SafeRLHF | F1: 0.926 -> 0.921 | Hurt |
| Use RSafe rather than LlamaGuard3-8B | Safety moderation · OpenAI Moderation | accuracy: 0.871 -> 0.817 | Hurt |
| Use RSafe rather than LlamaGuard3-8B | Safety moderation · XSTest | F1: 0.931 -> 0.816 | Hurt |
Surprising findings
- RSafe without RL (w/o rl) performs notably worse, showing RL alignment is needed beyond prompting alone.
- On JailbreakBench edge cases (17 prompts), RSafe underperforms GPT-4o (0.706 vs 0.882 accuracy).
Limitations
- Reasoning ability is bounded by the capacity of the backbone since no SFT cold start is used.
- Custom policy benefits depend on the user's ability to specify the relevant safety taxonomy.
- Weaker than GuardReasoner-8B and LlamaGuard3 on some metrics and on JailbreakBench edge cases.
Experimental setup
Datasets: ToxicChat Aegis OR-Bench BeaverTails SafeRLHF
Benchmarks: ToxicChat OpenAI Moderation AegisSafetyTest SafeRLHF BeaverTails XSTest WildGuardTest MedSafetyBench CARES JailbreakBench JailJudge
Metrics: accuracy F1
Baselines: OpenAI Moderation GPT-4o ShieldGemma-9B LlamaGuard3-8B WildGuard-7B GuardReasoner-8B
Builds on: GRPO DeepSeek-R1 GuardReasoner VERL
Closest papers by meaning
- $R^2$-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning (ICLR 2025 · similarity 0.91)
- Safety Pretraining: Toward the Next Generation of Safe AI (NeurIPS 2025 · similarity 0.91)
- STAIR: Improving Safety Alignment with Introspective Reasoning (ICML 2025 · similarity 0.91)
- Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements (ICLR 2025 · similarity 0.90)
- UpSafe℃: Upcycling for Controllable Safety in Large Language Models (NeurIPS 2026 · similarity 0.90)
- Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.