Research map / Large language models / Instruction and fine-tuning
WildChat: 1M ChatGPT Interaction Logs in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases WildChat, a corpus of about 1M opt-in user-ChatGPT conversations collected via free public GPT-3.5/GPT-4 chat services, analyzes its diversity, languages and toxicity, and fine-tunes Llama-2 7B into WildLlama as a demo.
Problem
Public datasets of real-world user-chatbot interactions are scarce; most instruction-tuning data is either proprietary, expert-curated, or model-generated, and existing natural logs are rarely released.
Approach
Collect real multi-turn, multilingual ChatGPT/GPT-4 conversations from free public chatbot services with explicit opt-in consent, anonymize PII, release them with coarse geography and hashed IPs and request headers, and show the data is useful for instruction tuning.
Key results
- WildLlama achieves an MT-bench average Likert score of 6.35, above Vicuna (6.13) and Llama-2 Chat (6.26) among 7B open models but well below GPT-4 (8.99).
- About 10.46% of WildChat user turns are flagged toxic by Detoxify or OpenAI Moderation, versus 3.08% for LMSYS-Chat-1M user turns under Moderation.
- English accounts for 52.94% of WildChat turns, with Chinese at 13.38% and Russian at 11.61%, covering 68 languages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama-2 7B on WildChat | Llama-2 7B · MT-bench | average Likert score: 6.26 -> 6.35 | Helped |
| Fine-tune Llama-2 7B on WildChat rather than use Vicuna | Llama-2 7B · MT-bench | average Likert score: 6.13 -> 6.35 | Helped |
| Fine-tune Llama-2 7B on WildChat rather than use Llama-2 Chat | Llama-2 7B · MT-bench | second-turn Likert score: 6.12 -> 5.90 | Hurt |
| Fine-tune Llama-2 7B on WildChat rather than use Llama-2 Chat | Llama-2 7B · MT-bench | first-turn Likert score: 6.41 -> 6.80 | Helped |
| Fine-tune Llama-2 7B on WildChat rather than use Vicuna | Llama-2 7B · MT-bench | WildLlama win rate against Vicuna: 10.00 -> 30.94 | Helped |
Surprising findings
- Chatbot turns were more toxic than user turns in April and May 2023, a trend that reversed after a June 2023 OpenAI model update.
- Jailbreak prompts copied from online communities reached success rates up to 71.16% (JailMommy) by the flagging proxy.
Limitations
- User population skews toward the IT/Hugging Face community and Reddit-referred users, so it may not represent the general population.
- Anonymity may induce selection bias toward more toxic content.
- Evaluation of WildLlama relies on MT-bench with GPT-4 as judge, and WildLlama lacks RLHF, underperforming Llama-2 Chat.
Experimental setup
Datasets: WildChat Alpaca Open Assistant Dolly ShareGPT LMSYS-Chat-1M MT-bench
Benchmarks: MT-bench WildChat toxicity analysis
Metrics: Likert score win/tie/loss rate negative log-likelihood toxicity rate jailbreak success rate entropy/embedding coverage
Baselines: Vicuna Llama-2 Chat GPT-3.5 GPT-4 LMSYS-Chat-1M ShareGPT Open Assistant Alpaca Dolly
Builds on: Vicuna MT-bench
Closest papers by meaning
- WildChat-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training (ICML 2025 · similarity 0.90)
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (ICLR 2024 · similarity 0.87)
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.85)
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models (NeurIPS 2024 · similarity 0.85)
- ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates (AAAI 2025 · similarity 0.82)
- First-Person Fairness in Chatbots (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.