Research map / Large language models / Reasoning and chain of thought

G-Zero: Self-Play for Open-Ended Generation from Zero Data

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes G-Zero, a verifier-free co-evolutionary self-play framework where a GRPO-trained Proposer is rewarded by Hint-δ (per-token log-likelihood shift from a self-generated hint) and a DPO-trained Generator learns from hint-assisted vs unassisted response pairs.

Problem

Self-evolution of LLMs works in verifiable domains but relies on proxy LLM judges in open-ended tasks, causing capability bottlenecks and reward hacking.

Approach

Uses Hint-δ, the per-token mean log-likelihood shift of the Generator's own unassisted response when conditioned on a self-generated hint, as an intrinsic reward for a Proposer (GRPO) and as a filter for DPO preference pairs (chosen = hint-assisted, rejected = unassisted), with no external judge or verifier.

Key results

• On Qwen3-8B-Base, G-Zero round 2 raises average over 7 metrics from 33.95 to 35.43 and AIME25 from 7.19 to 12.40, while R-Zero drops average to 32.59.
• On Llama-3.1-8B-Instruct, G-Zero round 2 raises AlpacaEval LC from 24.12 to 27.86 (+3.74) and average from 42.77 to 43.90.
• Lower-half δ filter ([0,50]) gives average 34.96 vs 34.65 for no filter and 34.04 for upper-half [50,100] on Qwen3-8B-Base round 1.

Surprising findings

• Hint-δ median increases from Round 1 to Round 2 even though the Generator is stronger, interpreted as a Proposer-Generator arms race.
• Math gains transfer from non-verifiable writing/advice training data, which make up over 70% of the DPO pool, while math is under 10%.

Limitations

• Gains are modest (e.g., average +1.5 points on Qwen3-8B-Base), and R-Zero comparison and some metrics regress in round 2 (AIME24, AlpacaEval LC on Qwen3).
• Theory covers only an idealized standard-DPO linear Bradley-Terry variant, not the length-normalized DPO used in practice.
• Hyperparameter sensitivity of δ filter band shows modest differences; results rely on AlpacaEval judged by an LLM judge.

Experimental setup

Datasets: self-generated question-hint pairs

Benchmarks: AIME24 AIME25 IFEval AlpacaEval 2.0

Metrics: mean@32 accuracy prompt-level strict/loose accuracy instruction-level strict/loose accuracy length-controlled win rate

Baselines: R-Zero base model

Builds on: R-Zero DPO GRPO LoRA

Closest papers by meaning

• Meta$^n$: Recursive Self-Improvement through Emergent Depth (NeurIPS 2026 · similarity 0.87)
• Nudging the Boundaries of LLM Reasoning (ICLR 2026 · similarity 0.86)
• ReVeal: Self-Evolving Code Agents via Reliable Self-Verification (ICLR 2026 · similarity 0.86)
• SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks (NeurIPS 2026 · similarity 0.86)
• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization (NeurIPS 2026 · similarity 0.85)
• Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.85)