Research map / Large language models / Alignment and preferences

Improving Data Efficiency via Curating LLM-Driven Rating Systems

ICLR 2025 · Large language models › Alignment and preferences · approach: LLM-as-judge and feedback data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DS2, a diversity-aware score curation method that corrects LLM-based quality ratings via a score transition matrix estimated from k-NN agreement and selects diverse high-quality instruction data, achieving strong results with 3.3% of the data.

Problem

LLM-based data quality ratings for instruction tuning contain systematic errors and biases, and selecting data by raw scores without accounting for these errors and diversity yields suboptimal instruction-tuned models.

Approach

Model LLM rating errors as a score transition matrix estimated from 2-NN embedding-cluster consensus statistics (without ground truth), use it to identify and correct misrated samples with rebalancing, and combine curated scores with a k-NN long-tail diversity score for selection.

Key results

• With LLaMA-3.1-8B and 10K selected samples, DS2 (rated by LLaMA-3.1-8B-Instruct) averages 60.2 on the OpenLLM leaderboard tasks versus 57.7 for full 300K data fine-tuning.
• With GPT-4o-mini ratings, score curation raises DS2 average from 60.2 to 61.4 on LLaMA-3.1-8B.
• At 1K samples, DS2 (GPT-rated) averages 54.8 on LLaMA-3.1-8B versus 50.2 for LIMA.

Measured effects reported in the paper

Surprising findings

• Only 3.3% of the data (10K of 300K) outperforms full-data fine-tuning, supporting that redundant low-quality samples can hurt performance.
• After score curation, weaker rating models (LLaMA, Mistral) match or surpass GPT-4o-mini's uncurated rating performance.

Limitations

• Sample-independent transition matrix assumption may be too strong.
• k-NN clusterability may be violated for text where small wording changes flip meaning; depends on embedding quality.
• Experiments limited to 7B/8B models and GPT-4o-mini as the GPT rater; unclear for larger models or GPT-4.

Experimental setup

Datasets: Flan V2 Open-Assistant 1 WizardLM Dolly Stanford Alpaca LIMA

Benchmarks: MMLU TruthfulQA GSM BBH TydiQA Vicuna-Bench MT-bench

Metrics: exact match F1 win-tie-loss rate

Baselines: Random Selection Completion Length Perplexity k-NN LESS AlpaGasus DEITA Full data (300K) LIMA

Builds on: LESS AlpaGasus DEITA label noise transition matrix estimation with clusterability

Closest papers by meaning

• DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.89)
• Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (ICLR 2025 · similarity 0.89)
• Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.89)
• Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
• Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.88)
• NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric (ICML 2025 · similarity 0.88)