Research map / Large language models / Reasoning and chain of thought

Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Generative Adversarial Reasoner (GAR), which jointly trains an LLM reasoner and an LLM-based slice-level discriminator with GRPO to provide dense step-level rewards for math reasoning.

Problem

LLM reasoners make process errors (wrong calculations, invalid steps), and sparse exact-match outcome rewards give weak credit assignment, while PRMs are costly and static LLM critics are noisy.

Approach

Co-evolve the reasoner and a discriminator adversarially: the discriminator scores logically complete reasoning slices (binary soundness plus short rationale, capped at 128 tokens) and is trained with GAN-style and alignment rewards, while the reasoner gets exact-match plus mean slice-level reward under GRPO.

Key results

• AIME24 improves from 54.0 to 61.3 for DeepSeek-R1-Distill-Qwen-7B (+7.3).
• AIME24 improves from 43.7 to 53.7 for DeepSeek-R1-Distill-Llama-8B (+10.0).
• Ablation: trainable discriminator with both alignment and discriminative rewards reaches 61.3 AIME24 vs 56.3 for standard RL and 58.6 for a fixed GAR discriminator.

Surprising findings

• Training with partial traces (three slices) and no final-answer reward reached 57.7 AIME24, above standard RL at 56.3, in 6 hours versus 16.
• Human experts' success distinguishing generated reasoning from Gemini traces dropped from 82.3% to 55.9% after GAR training on style-only discriminator data.

Limitations

• Balancing discriminator reasoning depth with compute efficiency remains challenging; longer analysis is costly.
• Slice-level dense rewards are averaged into a single trajectory-level signal, which may dilute local credit assignment and increase variance.
• Llama setup uses the 8B model as both reasoner and discriminator because no smaller compatible Llama reasoning model exists.

Experimental setup

Datasets: OpenR1-Math-220k S1K-1.1 CodeForces-CoT

Benchmarks: AIME24 AIME25 MATH500 GSM8K AMC23 OlympiadBench LiveMathBench-Hard LiveCodeBench HumanEval HumanEval+

Metrics: Pass@1 accuracy averaged over 30 samples

Baselines: Standard RL (GRPO exact-match) Fixed standard critic KL Cov Math-Shepherd

Builds on: GRPO GAN DeepSeek-R1-Distill OpenR1 KL Cov

Closest papers by meaning

• General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.93)
• SABER: Switchable and Balanced Training for Efficient LLM Reasoning (AAAI 2026 · similarity 0.91)
• Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning (NeurIPS 2026 · similarity 0.90)
• Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning (ICLR 2026 · similarity 0.90)
• SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
• QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation (ICLR 2026 · similarity 0.89)