Research map / Large language models / Language, safety and interpretability
WildChat-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WildChat-50M, a large synthetic chat dataset of responses from 50+ open-weight models to WildChat prompts, and uses it to compare data-generating models for SFT and build the Re-Wild SFT mix.
Problem
Large-scale comparative analysis of synthetic data generating models (DGMs) and LLM judges for post-training is hard because publicly available large-scale synthetic chat datasets are scarce.
Approach
Re-generates responses to existing WildChat prompts with over 50 open-weight models (0.5B to 104B), creating a public resource for controlled comparisons of DGM quality, and builds Re-Wild, an SFT mix that uses WildChat-Qwen2.5-72B responses plus MMLU auxiliary and Tulu 3 persona algebra data.
Key results
- Re-Wild SFT mix outperforms the Tulu 3 SFT mixture with only 40% as many samples, per the abstract and spider-chart comparison.
- Choice of DGM has large effect: Qwen2.5-72B-generated SFT reaches AlpacaEval LC 41.00 versus 21.12 for Llama-3.1-8B-Instruct-generated SFT.
- Fine-tuned models do not reliably inherit DGM benchmark strength: agreement rate with DGM rankings is at chance (0.5) across six benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Re-Wild instead of the Tulu-3 SFT mix, with 40% as many samples | Llama-3.1-8B-Base SFT · Post-training benchmarks | aggregate performance: 40% as many samples | Helped |
| Increase SFT data from 100k to 250k to 500k samples (ablation) | SFT models trained on WildChat responses · Avg across benchmarks | Avg: 100k, 250k and 500k | Helped |
| Truncate Qwen2.5-72B responses to the Llama-3.3-70B response-length limit (ablation) | Llama-3.1-8B-Base SFT on Qwen2.5-72B responses · 9 benchmarks | average score: .400 -> .404 | Helped |
| Blend responses from two DGMs rather than using a single DGM (ablation) | Llama-3.1-8B-Base SFT · SFT benchmarks | Avg | No clear effect |
| Use a DGM from the same or a highly similar model family as the SFT target (ablation) | Qwen2-7B and Llama-3-8B SFT targets · Post-training benchmarks | benchmark performance | Helped |
| Use Qwen2.5-72B-Instruct rather than Llama-3.3-70B as the DGM (ablation) | Llama-3.1-8B-Base SFT · MTBench | score: 6.38 -> 6.72 | Helped |
| Use Qwen2.5-72B-Instruct rather than Llama-3.3-70B as the DGM (ablation) | Llama-3.1-8B-Base SFT · AlpacaEval | score: 24.91 -> 41.00 | Helped |
Surprising findings
- Responses from 50+ different models are unusually similar to one another in ROUGE and METEOR scores.
- Blending responses from multiple DGMs gives no benefit over the average of their individual performance.
Limitations
- Only SFT was studied; other post-training methods such as preference optimization were not evaluated.
- Benchmarks do not cover specialized domains such as coding or legal reasoning.
- Dataset access requires approval and the WildChat user prompts retain possible personal information.
Experimental setup
Datasets: WildChat-1M MMLU auxiliary train Tulu 3 Persona Hub Algebra
Benchmarks: MTBench AlpacaEval 2 BBH GPQA MATH MUSR IFEval MMLU Pro MixEval
Metrics: length-controlled win rate prompt-level strict accuracy accuracy average score preference rate
Baselines: Tulu 3 SFT mix Magpie Align SFT UltraChat
Builds on: WildChat Tulu 3 Evalchemy vLLM
Closest papers by meaning
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (ICLR 2024 · similarity 0.91)
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.90)
- WildChat: 1M ChatGPT Interaction Logs in the Wild (ICLR 2024 · similarity 0.90)
- OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (ICLR 2024 · similarity 0.89)
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.88)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.