Research map / Large language models / Instruction and fine-tuning
WizardArena: Post-training Large Language Models via Simulated Offline Chatbot Arena
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WizardArena, a simulated offline chatbot arena judged by open-source Llama3-70B-Chat, and Arena Learning, an iterative battle-based data flywheel that trains WizardLM-β (Mistral-7B) with SFT, DPO, and PPO.
Problem
Human-based chatbot arenas and manual instruction-data curation for LLM post-training are expensive, slow, and limited in scale, and most models cannot join online arenas to guide development.
Approach
Replace human annotators with an open-source LLM judge to run pairwise offline battles on a curated diverse+hard test set, producing Elo rankings that track LMSYS Chatbot Arena; reuse the same battles to select weakness-targeted training data (pair-judge) for SFT, DPO, and PPO in iterative rounds.
Key results
- WizardArena-Mix achieves 92.80% average consistency with LMSYS Chatbot Arena, close to Arena-Hard-v1.0 at 91.57%, while MT-Bench scores 33.02%.
- Over three iterations, WizardLM-β ELO on WizardArena rises from 875 (SFT-I0) to 1274 (PPO-I3) and MT-Bench from 6.41 to 7.81.
- Pair-judge data selection beats original 30k data by 22 Elo points on Offline-Mix at 10k samples in SFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on Arena Learning SFT data rather than the initial SFT setup | WizardLM-β · WizardArena-Mix | ELO: a 188-point rise | Helped |
| Train on Arena Learning SFT data rather than the initial SFT setup | WizardLM-β · MT-Bench | score: a 0.57-point increase | Helped |
| Add DPO after SFT | WizardLM-β · WizardArena-Mix | ELO: 135 points | Helped |
| Add PPO after SFT | WizardLM-β · WizardArena-Mix | ELO: 142 points | Helped |
| Add PPO after SFT and DPO (ablation) | WizardLM-β · MT-Bench | score: a 0.05-point improvement | Helped |
| Add PPO after SFT and DPO (ablation) | WizardLM-β · WizardArena-Mix | ELO: 21 points | Helped |
| Select SFT instructions where the base model loses and use the winning response (ablation) | WizardLM-β SFT · WizardArena-Mix | ELO: 22-point improvement | Helped |
| Select SFT instructions where the base model loses and use the winning response (ablation) | WizardLM-β SFT · MT-Bench | score: 0.17 points over Original | Helped |
| Replace pair-judge SFT selection with INSTAG selection (ablation) | WizardLM-β SFT · WizardArena-Mix | ELO: 11 points | Hurt |
| Replace pair-judge SFT selection with INSTAG selection (ablation) | WizardLM-β SFT · MT-Bench | score: 0.06 points | Hurt |
| Use SFT data selected by pair-judge instead of original D1 data (ablation) | WizardLM-β SFT · WizardArena-Mix | ELO: 22-point improvement | Helped |
| Use SFT data selected by pair-judge instead of original D1 data (ablation) | WizardLM-β SFT · MT-Bench | score: 0.17 points over Original | Helped |
Surprising findings
- MT-Bench shows large ranking fluctuations; GPT-3.5-Turbo-0613 and DeepSeek-LLM-67B-Chat score well on MT-Bench but rank low on LMSYS Arena.
- Llama3-70B-Chat judge agrees with GPT-4 judge at 92.14% average consistency, suggesting open judges can replace GPT-4.
Limitations
- If the judge model fails to imitate human evaluators, rankings and training data may be compromised.
- Models may still generate unethical or misleading information.
Experimental setup
Datasets: Alpaca FLAN LMSYS-Chat-1M OpenOrca WizardLM ShareGPT
Benchmarks: LMSYS Chatbot Arena EN MT-Bench WizardArena Offline-Diverse WizardArena Offline-Hard WizardArena Offline-Mix OpenLLM Leaderboard GSM8k AlpacaEval 2.0
Metrics: Elo score Spearman correlation human agreement differentiation MT-Bench score win rate average accuracy
Baselines: Command R+ Qwen1.5-72B-Chat Qwen1.5-32B-Chat WizardLM-70B-v1.0 Tulu-2-DPO-70B OpenChat-3.5 GPT-3.5-Turbo-0613 Zephyr-7b-alpha
Builds on: Chatbot Arena MT-Bench DPO PPO WizardLM
Closest papers by meaning
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.88)
- WildChat-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training (ICML 2025 · similarity 0.87)
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.87)
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (ICML 2024 · similarity 0.86)
- From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline (ICML 2025 · similarity 0.86)
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.