Research map / Large language models / Reasoning and chain of thought

Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Intervened Preference Optimization (IPO), which replaces the first compliance cue in unsafe reasoning traces with a safety trigger and runs DPO on the resulting prefix-sharing preference pairs to make large reasoning models reason safely.

Problem

Large reasoning models often produce harmful content in their chain-of-thought even when final responses are safe, and RL with safety rewards (GRPO) gives weak signal because rollouts lack safety diversity.

Approach

Identify safety triggers (sentences that secure safe continuation) and compliance cues (sentences that predict unsafe continuation); intervene at the first compliance cue by inserting a safety trigger, then apply DPO on the divergent pairs so the model prefers the corrected continuation, with an auxiliary SFT loss and an over-refusal mitigation stage.

Key results

• On WildJailbreak, IPO reduces DS-8B reasoning harmfulness from 82.4% to 23.4%, versus 36.3% for the best baseline (GRPO).
• IPO yields an average reasoning harmfulness of 15.3% on DS-8B against 18.5% for GRPO, and 18.4% on DS-7B against 24.7% for GRPO.
• Qwen3-8B average reasoning harmfulness drops to 13.9% from 51.3% base, with average reasoning ability rising from 79.5% to 80.2%.

Surprising findings

• RealSafe achieves lower response harmfulness than IPO largely through over-conservativeness, with lower XsTest compliance.
• IPO's DS-8B XsTest compliance (80.0%) is lower than the base model's 98.4%, indicating a mild over-refusal trade-off.

Limitations

• Over-refusal increases on benign prompts after safety alignment (XsTest compliance drops on some models).
• Compliance cue detection relies on GPT-4o with about 83% agreement with human annotation.

Experimental setup

Datasets: STAR-1

Benchmarks: JailbreakBench StrongReject WildJailbreak XsTest AIME2024 MATH-500 GPQA-Diamond HumanEval

Metrics: harmful ratio (reasoning and response) compliance rate pass@1

Baselines: SafeChain RealSafe STAR SafeKey GRPO TARS BackTrack

Builds on: DPO GRPO STAR-1 RPO

Closest papers by meaning

• SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment (NeurIPS 2025 · similarity 0.94)
• Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models (NeurIPS 2026 · similarity 0.93)
• Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away (ICML 2026 · similarity 0.92)
• Internalizing Safety Understanding in Large Reasoning Models via Verification (ICML 2026 · similarity 0.92)
• STAIR: Improving Safety Alignment with Introspective Reasoning (ICML 2025 · similarity 0.91)
• AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models (ICLR 2026 · similarity 0.90)