Research map / Large language models / Reasoning and chain of thought
Reasoning as an Adaptive Defense for Safety
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TARS, an RL recipe (lightweight SFT warmstart, mixed harmful/harmless/ambiguous prompts, split safety and helpfulness rewards) that trains small reasoning models to adaptively reason about safety and reduce over-refusal.
Problem
Safety training of LLMs via reasoning often yields degenerate over-refusal, and it is unclear how to design data and rewards so that test-time reasoning improves the safety-refusal trade-off and robustness to jailbreaks.
Approach
Use GRPO on long chain-of-thought with a lightly SFT-warmed-up Qwen-2.5 model, mixing harmful prompts (scored by a moderation API safety reward) with harmless and ambiguous prompts (scored by a preference reward model), so the model reasons per prompt instead of refusing shortcut-style.
Key results
- TARS 1.5B models achieve better safety-refusal trade-off than 7-8B models including Llama-RR, Mistral-RR, Llama-8B, and SafeChain, and the 7B TARS model outperforms all.
- Under GCG with properly formatted reasoning, TARS reaches 92.79% DSR vs 82.73% for SFT; under PAIR with EOT present, 94.45% vs 72.97%.
- Reasoning length on Sorry-Bench varies by category, from about 290 tokens for hate speech generation to about 457 tokens for unqualified advice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with RL and reasoning rather than supervised reasoning traces | Qwen-2.5-1.5B-Instruct; TARS vs. SFT · GCG, responses with EOT | DSR (%): 82.73% > 92.79% | Helped |
| Train with RL and reasoning rather than supervised reasoning traces | Qwen-2.5-1.5B-Instruct; TARS vs. SFT · PAIR, responses with EOT | DSR (%): 72.97% > 94.45% | Helped |
| Train with RL and reasoning rather than supervised reasoning traces | Qwen-2.5-1.5B-Instruct; TARS vs. SFT · GCG, broken-format responses | DSR (%): 78.90% > 83.82% | Helped |
| Use lightweight SFT warmstart instead of heavy SFT (ablation) | TARS · Safety-refusal trade-off | safety-refusal trade-off | Helped |
| Use heavy SFT with learning rate 1×10⁻⁴ for 6 epochs instead of lightweight SFT (ablation) | TARS · Safety-refusal trade-off | safety-refusal trade-off | Hurt |
| Add harmless prompts and task-completion rewards to RL (ablation) | TARS · Harmless and ambiguous prompts | reasoning length | Helped |
| Train on only harmful prompts with a safety reward (ablation) | TARS RL training · Harmless prompts | refusal | Hurt |
| Add ambiguous OR-Bench prompts rewarded with GRM (ablation) | TARS · Ambiguous prompts | refusal rate | Helped |
| Use separate safety and helpfulness rewards instead of one GRM preference reward (ablation) | TARS · Safety-refusal trade-off | safety-refusal trade-off | Helped |
| Use GRM directly on harmful prompts instead of mixing harmless prompts with separate rewards (ablation) | TARS RL training · Safety-refusal trade-off | safety-refusal trade-off | Mixed |
| Mix harmless prompts with a separate helpfulness reward rather than use GRM directly on harmful prompts (ablation) | TARS RL training · Safety-refusal trade-off | safety-refusal trade-off | Helped |
| Use RL with reasoning rather than RL without reasoning | TARS vs. RL · Safety-refusal trade-off | safety-refusal trade-off | Helped |
Surprising findings
- RL trained only on harmful prompts with a safety reward quickly collapses reasoning length and causes over-refusal on harmless prompts (reward hacking).
- SFT with reasoning traces overfits and is insensitive to the harmful/harmless mixture ratio, while RL without reasoning outperforms SFT with reasoning.
Limitations
- Relies on a single base family (Qwen-2.5) and a small number of prompts (2000 RL prompts).
- Safety reward uses an external moderation API and a preference reward model, which may bias results.
- GCG attacks target the reasoning segment rather than the answer, so attack strength against answer-targeted optimization is not fully evaluated.
Experimental setup
Datasets: WildJailbreak Aegis AI Content Safety Dataset 2.0 SafeEdit Rainbow Teaming UltraFeedback OR-Bench
Benchmarks: HarmBench XSTest WildChat Sorry-Bench
Metrics: defense success rate (DSR%) refusal rate via StrongReject reasoning and response length SVM margin on UMAP embeddings
Baselines: Llama-RR Mistral-RR RealSafe-R1 SafeChain Deliberative Alignment Llama-3.1-8B-Instruct Llama-3.2-1B-Instruct SFT with reasoning DPO (RPO) RL without reasoning
Builds on: GRPO DeepSeek-R1 Circuit breakers (representation re-routing) RPO Deliberative Alignment
Closest papers by meaning
- Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away (ICML 2026 · similarity 0.90)
- Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention (ICLR 2026 · similarity 0.90)
- Internalizing Safety Understanding in Large Reasoning Models via Verification (ICML 2026 · similarity 0.89)
- AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.89)
- RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.89)
- Safety Reasoning with Guidelines (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.