Research map / Large language models / Reasoning and chain of thought

Learning Robust Reasoning through Guided Adversarial Self-Play

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GASP, a self-play RLVR method where one model acts as polluter (corrupting clean reasoning windows) and agent (recovering under corruption), with in-distribution self-generated repair guidance, improving robustness across four 1.5B-8B reasoning models.

Problem

Strong reasoning models trained with RLVR fail when the conditioning context is fallible (corrupted CoT, misleading partial solutions, perturbed inputs), since outcome-only RL optimizes only clean-context correctness.

Approach

A single model plays two roles in adversarial self-play: a polluter learns locally coherent corruptions that induce failure, and an agent learns to detect and repair under them, using only terminal correctness rewards; in-distribution repair guidance imitates self-generated repairs to counter recovery-signal scarcity, instead of off-distribution teacher (GPT-5) fixes.

Key results

• GASP raises average robustness score on Qwen3-8B (LoRA) from 27.3 initial to 83.4, versus 53.2 for GASP-T teacher-guided variant.
• On R1-Distill-Qwen-1.5B, GASP reaches 95.6% recoverability on GSM8K under GPT-5 polluted context, up from 12.4% initial checkpoint.
• Self-guided GASP keeps representation drift smaller than teacher-guided GASP-T and better preserves TruthfulQA accuracy in the paper's analysis.

Surprising findings

• Self-generated in-distribution repairs outperform GPT-5 teacher repairs as guidance targets, which the authors attribute to lower likelihood of teacher fixes under the student policy.

Limitations

• Robustness gains are measured mainly against GPT-5-generated polluters and self-generated corruptions, with held-out polluter evaluation only partly covering other attack distributions.
• Theoretical analysis relies on near-orthogonality assumptions for score-function cross terms.

Experimental setup

Datasets: MATH GSM8K MR-GSM8K MHPP RUPBench CommonsenseQA TruthfulQA

Benchmarks: GSM8K MR-GSM8K MHPP RUPBench GSM8K RUPBench CommonsenseQA

Metrics: pass@1 accuracy ACCreason self-revision accuracy representation PCA shift

Baselines: Initial checkpoint ASP GASP-T GASP-p-fixed GASP-p-gpt-5

Builds on: GRPO SPIN SCoRe STaR

Closest papers by meaning

• Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning (NeurIPS 2026 · similarity 0.88)
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data (NeurIPS 2025 · similarity 0.87)
• SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning (NeurIPS 2025 · similarity 0.87)
• AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models (ICLR 2026 · similarity 0.86)
• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.86)
• Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards (NeurIPS 2025 · similarity 0.86)