Research map / Large language models / Reasoning and chain of thought

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Defines DSAR, a metric for reasoning-answer safety inconsistency in large reasoning models, and proposes SARA, an RL method that rewards early safety-aware reasoning plus safe final answers, improving consistency under prefilling attacks while preserving helpfulness and utility.

Problem

RL for large reasoning models rewards only final answers, so reasoning traces and answers can convey contradictory safety signals (deceptive safety alignment), which is amplified by prefilling attacks; no direct metric existed for this.

Approach

Introduces DSAR, which judges sentence-level safety awareness in the CoT plus full-trace and answer safety, and SARA, a DAPO-based RL method whose reward multiplies full-trace safety by an earliness-weighted safety-awareness score and adds an answer safety term for harmful prompts, with a refusal-based reward for benign prompts.

Key results

• SARA raises SAR under adversarial prefilling on StrongReject from 57.80 (RECAP) to 75.10 on DS-Qwen3-8B, with 1-DSAR of 85.30.
• SARA achieves the best Avg. harmonic score of 82.76 on DS-Qwen3-8B and 81.97 on DS-Qwen2-14B, versus 77.61 and 81.67 for RECAP.
• Under standard prompting on SafeChain, DS-Qwen3-8B 1-DSAR rises from 85.00 (original) to 96.40 with SARA.

Surprising findings

• Prefilling degrades reasoning safety (SAR) more sharply than final-answer safety, so answers can look superficially safe while reasoning remains unsafe.
• Counter-aligned prefill augmentation improves prefilling robustness but lowers SARA's standard-prompting SAR, SS, and 1-DSAR on both models.

Limitations

• Evaluation relies on LLM judges (GPT-oss-safeguard, Qwen3Guard) validated on only 100 human-annotated generations, with few cases in the inconsistency categories.
• SARA trades some final-answer safety (SS) for reasoning safety awareness, e.g., SS drops from 99.40 to 97.44 under prefilling versus RECAP.
• Experiments are limited to two model sizes (8B and 14B) and a 2K-prompt training set.

Experimental setup

Datasets: SafeChain FalseReject

Benchmarks: StrongReject SafeChain ICL attack H-CoT OR-Bench-hard Fortress XSTest GSM8K MMLU-Pro

Metrics: SAR SS DSAR helpfulness score accuracy harmonic mean

Baselines: SafeChain SafePath STAR-1 RECAP

Builds on: DAPO RECAP GRPO SafeChain SafePath

Closest papers by meaning

• AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.93)
• Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention (ICLR 2026 · similarity 0.93)
• Internalizing Safety Understanding in Large Reasoning Models via Verification (ICML 2026 · similarity 0.92)
• Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representations (ICML 2026 · similarity 0.91)
• AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models (ICLR 2026 · similarity 0.90)
• SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment (NeurIPS 2025 · similarity 0.90)