Research map / Large language models / Instruction and fine-tuning

OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

ICLR 2024 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 27 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OpenChat, which fine-tunes LLaMA-2-13B on mixed-quality ShareGPT data using Conditioned-RLFT (C-RLFT), treating GPT-4 vs GPT-3.5 sources as coarse rewards and class-conditioned prompts, solved via RL-free reward-weighted supervised learning.

Problem

SFT treats all training data equally despite mixed quality, while RLHF/RLFT methods need costly pairwise or ranking preference labels. The paper asks how to use mixed-quality SFT data without preference labels.

Approach

Treat data source (GPT-4 vs GPT-3.5) as a coarse reward class, condition the policy on class-specific prompt templates, and regularize toward a class-conditioned reference policy. The KL-regularized optimum reduces to reward-weighted regression, which is solved with single-stage supervised learning and no RL or preference labels.

Key results

• openchat-13b reaches 77.3 average win rate across AlpacaEval, MT-bench and Vicuna-bench, the highest among 13b open-source models in the paper's table.
• Ablations: removing the class condition drops the average from 77.3 to 60.5; removing the reward weighting gives 74.0.
• On AGIEval zero-shot, openchat-13b averages 36.4 versus 35.0 for the base llama-2-13b, the only fine-tuned model above base.

Measured effects reported in the paper

Surprising findings

• Only GPT-4 data (about 6k conversations) SFT scores 64.5 average, far above SFT on the full 70k ShareGPT mix (52.7), suggesting data quality matters more than quantity.
• Using the GPT-3.5 conditioning prompt at inference instead of the GPT-4 prompt causes a substantial performance drop.

Limitations

• The coarse reward assumes data source equals quality, which may be overly simplistic; per-sample quality is not modeled.
• Focus is on instruction following; reasoning ability is not addressed and is left as future work.
• Evaluation relies mostly on GPT-4-based automatic judges, with a self-enhancement bias concern only partly addressed by also using gpt-3.5 and claude-2.

Experimental setup

Datasets: ShareGPT

Benchmarks: AlpacaEval MT-bench Vicuna-bench AGIEval

Metrics: win rate score accuracy

Baselines: vicuna-v1.5-13b wizardlm-v1.2-13b llama-2-chat-13b ultralm-13b gpt-3.5-turbo gpt-4 claude llama-2-chat-70b

Builds on: KL-regularized RL RLHF DPO Vicuna goal-conditioned supervised learning

Closest papers by meaning

• WildChat-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training (ICML 2025 · similarity 0.89)
• Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data (ICML 2025 · similarity 0.88)
• Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models (AAAI 2025 · similarity 0.88)
• Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization (NeurIPS 2025 · similarity 0.87)
• Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.87)
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (ICML 2024 · similarity 0.87)