Research map / Large language models / Reasoning and chain of thought

Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representations

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CRAFT, which combines contrastive latent representation learning (LCLR) with GRPO using latent-semantic, textual-safety, and latent-textual consistency rewards to reduce superficial safety alignment in large reasoning models.

Problem

Large reasoning models can produce unsafe reasoning traces that leak harmful content even when final responses are safe refusals (superficial safety alignment), making jailbreak robustness incomplete.

Approach

Shapes hidden-state geometry of reasoning traces with contrastive prototype/invariance/calibration losses, then runs GRPO with rewards that push latent states toward the safe region and enforce agreement between latent safety predictions and the textual safety of the final response.

Key results

• CRAFT reaches average StrongReject-based reasoning/response scores of 0.062 on DeepSeek-R1-Distill-Llama-8B and 0.099 on Qwen3-4B-Thinking, the best averages versus baselines (IPO 0.084 and 0.130).
• Reports 82.1% average reasoning-safety improvement and 89.6% final-response safety improvement over base models.
• Reports an 8.0% average improvement in math and code performance over the base model, though this comparison is across mixed benchmarks and its basis is unclear.

Surprising findings

• CRAFT is not top-ranked on every setting; the paper notes it trails IPO or RealSafe in some DeepSeek-R1-Distill-Llama-8B cases.
• Removing the LCLR module causes the largest ablation drop (32.4%), indicating latent contrastive structure is essential.

Limitations

• GRPO-based latent optimization is computationally costly and training budgets were constrained.
• Latent safety heads and prototypes do not transfer across model families, requiring per-family training.
• Not validated against fully adaptive attacks targeting the latent safety mechanism.

Experimental setup

Datasets: R2D-R1 StrongReject JailbreakBench AIME 2024 Minerva MATH-500 LiveCodeBench

Benchmarks: JailbreakBench StrongReject AIME24 MATH-500 LiveCodeBench Minerva GPTFuzzer AutoDAN

Metrics: StrongReject score reasoning-trace safety rate pass@1

Baselines: SafeChain RealSafe STAR SafeKey IPO ReasoningShield

Builds on: GRPO SimCLR STAR R2D StrongReject

Closest papers by meaning

• Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models (NeurIPS 2026 · similarity 0.91)
• Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away (ICML 2026 · similarity 0.91)
• AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.90)
• Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention (ICLR 2026 · similarity 0.90)
• SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment (NeurIPS 2025 · similarity 0.89)
• SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization (CVPR 2026 · similarity 0.88)