Research map / Large language models / Reasoning and chain of thought

Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Construction

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GRGC, which adds a Gaussian groupwise optimal-transport calibration to the GAD critic and a signed power transform on group-normalized rewards before GRPO, improving black-box distillation of small students from GPT-5 and Doubao-Seed-2.0 teachers.

Problem

On-policy adversarial black-box distillation (GAD) trains a critic with a Bradley-Terry objective that does not control within-group reward dispersion, so the student-side reward groups used for GRPO advantages can collapse or have fragile margins.

Approach

Two coupled interventions on student reward groups: (1) critic-side Gaussian groupwise OT calibration that matches sorted, mean-centered rewards to Gaussian quantiles, and (2) policy-side signed power transform of standardized group rewards before computing GRPO advantages.

Key results

• On GPT-5 teacher with Qwen2.5-3B, GRGC reaches 50.19 LMSYS score and 45.3% win rate vs GAD's 48.44 and 25.1%.
• With Doubao-Seed-2.0 teacher on Llama-3.2-3B, GRGC gets 56.4% LMSYS win rate vs GAD's 45.3%.
• Low-dispersion groups (relative std <0.15) drop from about 8.5% under GAD to about 1.3-1.9% under GRGC over 1500 steps; overhead is 0.06-0.11% of step time.

Surprising findings

• Adding only the CGC term gives below-GAD results after one adversarial epoch on Qwen2.5-3B (47.52 vs 48.44), and gains appear mainly after the second epoch.
• The paper's toy analysis shows the BT critic loss can decrease as within-group reward dispersion collapses at fixed mean.

Limitations

• Gains after one adversarial epoch are inconsistent; CGC alone can underperform GAD early in training.
• Evaluation relies mainly on LLM-as-judge (Qwen2.5-72B) against reference answers; judge bias possible.
• Gaussian target is a chosen prior; alternatives are only partially compared.

Experimental setup

Datasets: LMSYS-Chat-1M subset Dolly-Train

Benchmarks: LMSYS held-out test (479 samples) DollyEval (500 samples) SelfInst (242) VicunaEval (80) IFEval Math500

Metrics: Qwen2.5-72B judge score win rate vs reference low-dispersion frequency top-2 advantage gap

Baselines: SeqKD GAD

Builds on: GAD GRPO SeqKD Bradley-Terry reward modeling Wasserstein distance

Closest papers by meaning

• GVPO: Group Variance Policy Optimization for Large Language Model Post-Training (NeurIPS 2025 · similarity 0.88)
• Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex (NeurIPS 2026 · similarity 0.87)
• Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation (NeurIPS 2026 · similarity 0.86)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.86)
• Improving Reward Model Generalization from Adversarial Process Enhanced Preferences (ICML 2025 · similarity 0.86)
• Adversaries With Incentives: A Strategic Alternative to Adversarial Robustness (ICLR 2025 · similarity 0.86)