Research map / Large language models / Language, safety and interpretability

Safety Reasoning with Guidelines

ICML 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Safety Reasoning with Guidelines (SRG), which synthesizes guideline-conditioned reasoning traces for unsafe queries, filters them by rejection sampling, and context-distills them into the model to improve OOD jailbreak generalization over refusal training.

Problem

Refusal training (RT) generalizes poorly to out-of-distribution jailbreak attacks; the paper asks whether models already hold latent safety knowledge that RT fails to elicit.

Approach

Use Best-of-N evaluation to show latent safety knowledge exists, then train models to produce explicit, guideline-aligned safety reasoning (with self-reflection and self-refinement) before refusing, internalized via context distillation so guidelines are removed at inference.

Key results

• SRG on LLaMA-3.1-8B reduces greedy-decoding ASR on OOD attacks by an average of 46.3% and on 70B by 54.8% versus refusal training, reaching near 0% on JailbreakChat and SelfCipher.
• Under BoN with N=64, SRG achieves approximately 0% ASR across all five attacks on the evaluated models.
• On PAIR, 70B-SRG with a role-awareness guideline added at inference reaches 4% ASR versus 94% for 70B-RT under greedy decoding.

Measured effects reported in the paper

Surprising findings

• Best-of-N sampling with N=128 reduces refusal-trained model ASR on OOD attacks by about 60% on average, suggesting latent safety knowledge exists but is not elicited by greedy decoding.
• Stronger reasoning teacher DeepSeek-R1-Qwen-32B gave weaker results with reasoning-only guidelines (18% illegal ASR) than GPT-4o, though extra guidelines helped.

Limitations

• Only rejection fine-tuning (SFT) was used; RL-based training was not explored due to computational constraints.
• SRG still lags LAT and RR on PastTense and Persuasive Attack in the comparison with existing safety training methods.
• Reasoning-based supervision lowers helpfulness only mildly but over-refusal on XSTest is 92% vs 95% for the instruct baseline, and helpfulness gains are mixed across tasks.

Experimental setup

Datasets: PKU-SafeRLHF UltraFeedback Do-Not Answer HarmBench JailbreakChat SelfCipher PastTense Persuasive Attack PAIR HumanEval MBPP GSM8K

Benchmarks: Do-Not Answer HarmBench JailbreakChat SelfCipher PastTense Persuasive Attack PAIR HumanEval MBPP GSM8K MATH BFCL

Metrics: attack success rate accuracy over-refusal rate

Baselines: Refusal Training LAT RR LLaMA-3-8B-Instruct GPT-4o

Builds on: Refusal Training Context Distillation Representation Engineering Deliberative Alignment

Closest papers by meaning

• Reasoning as an Adaptive Defense for Safety (NeurIPS 2025 · similarity 0.89)
• LLMs Encode Harmfulness and Refusal Separately (NeurIPS 2025 · similarity 0.89)
• RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.88)
• Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs (NeurIPS 2026 · similarity 0.88)
• $R^2$-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning (ICLR 2025 · similarity 0.88)
• Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMs (NeurIPS 2026 · similarity 0.88)