Research map / Large language models / Reasoning and chain of thought

AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AlphaAlign, a pure RL (PPO) framework that uses a verifiable refusal-based safety reward plus a normalized helpfulness reward to elicit proactive safety reasoning in LLMs without supervised safety reasoning data.

Problem

Safety alignment of LLMs often yields superficial refusal shortcuts, over-refusal, and utility loss, while reasoning-based alignment depends on intensive supervision or handcrafted safety specifications.

Approach

Treat safety as a verifiable property of the final answer (refusal vs. compliance matched against binary harmfulness labels), reward correctly formatted justified refusals and penalize over-refusal, and add a group-normalized, thresholded helpfulness reward for benign prompts so that safety reasoning emerges from RL alone.

Key results

• On Qwen2.5-3B-Instruct, AlphaAlign lowers PAIR ASR from 67.69% to 4.61% and GCG ASR from 49.04% to 0.77%, while raising CoCoNot over-refusal accuracy from 88.92% to 91.29%.
• AlphaAlign converges in under 200 RL steps and improves GSM8K by +4.4 points on Qwen2.5-3B-Instruct (74.3) while MMLU drops only 0.1.
• On Llama3.2-3B-Instruct, AlphaAlign reduces WildGuardTest ASR from 13.98% to 2.43%, but GSM8K drops 8.3 points (70.7).

Surprising findings

• Qwen2.5-3B base model trained with only the verifiable safety reward (AlphaAlign-Zero) shows substantial safety gains without any safety supervised data.
• Llama3.2-3B-Instruct loses 8.3 points on GSM8K after AlphaAlign, showing utility is not uniformly preserved.

Limitations

• Uses only binary safety labels and a string-matching refusal verifier, leaving richer verifiers unexplored.
• Only tested on models up to about 7B due to compute constraints.
• Harmful-prompt labels are assumed correct; mislabeled data could compromise defenses (noted in broader impacts).

Experimental setup

Datasets: SCoT harmful data Circuit Breaker dataset Dolly XSTest StrongREJECT AdvBench WildGuardTest JailbreakTrigger CoCoNot MMLU AlpacaEval GSM8K

Benchmarks: StrongREJECT AdvBench WildGuardTest JailbreakTrigger PAIR GCG CoCoNot MMLU AlpacaEval GSM8K

Metrics: ASR-% over-refusal accuracy-% MMLU accuracy AlpacaEval win rate GSM8K accuracy CKAS

Baselines: Direct Refusal Circuit Breaker SCoT

Builds on: PPO GRPO RLVR SCoT Circuit Breaker

Closest papers by meaning

• Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models (NeurIPS 2026 · similarity 0.93)
• BSO: Safety Alignment Is Density Ratio Matching (NeurIPS 2026 · similarity 0.92)
• Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.92)
• Internalizing Safety Understanding in Large Reasoning Models via Verification (ICML 2026 · similarity 0.91)
• SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization (CVPR 2026 · similarity 0.91)
• Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away (ICML 2026 · similarity 0.91)