Research map / Large language models / Instruction and fine-tuning

Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models

ICLR 2024 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Docta, a training-free framework that estimates label-noise transition matrices and data credibility from k-NN label consensus, then uses it to detect and fix label errors in 11 harmlessness/toxicity datasets.

Problem

Annotation errors and biases in safety-alignment datasets (toxicity and harmlessness classification) can cause harmful fine-tuned models, but manually re-annotating large datasets is prohibitively expensive.

Approach

Estimate the noise transition matrix without true labels using k-NN label clusterability and consensus statistics over neighboring examples, define a scalar data credibility from the transition matrix's distance to identity, and flag low-agreement instances via cosine-similarity scores with a Bayes-derived per-class threshold.

Key results

• Across 11 datasets built from five benchmarks, the framework finds and fixes an average of 6.16% label errors, from about 2% (cleanest) to over 15% (Anthropic Harmless).
• Fine-tuning BERT on Docta-cleaned Civil Comments raises toxicity F1 from 69.83 to 73.49 on i.i.d. test data.
• In a controlled human check, 415 of 604 human-verified label errors were flagged by the algorithm (68.71% hit rate).

Measured effects reported in the paper

Surprising findings

• Anthropic Harmless had very low initial data credibility (about 50), attributed to noise introduced by splitting pairwise conversations.
• Label cleaning improved labeling accuracy by about 5% even when initial accuracy was already above 90%.

Limitations

• Assumes transition matrix is locally constant given features and k-NN label clusterability, which may not hold broadly.
• Human verification hit rate is 68.71%, so many detected errors may be false positives and some errors are missed.
• Cleaning may introduce errors if applied to already accurate labels, as the ethics statement cautions.

Experimental setup

Datasets: Jigsaw Civil Comments PKU BeaverTails PKU SafeRLHF Anthropic Harmless Anthropic Red-Team

Benchmarks: Jigsaw Civil Comments PKU BeaverTails PKU SafeRLHF Anthropic Harmless Anthropic Red-Team

Metrics: data credibility estimated transition matrix F1-score test accuracy labeling accuracy hit rate cost reduction

Baselines: Raw training labels

Builds on: k-NN label clusterability transition matrix estimation for noisy labels co-teaching

Closest papers by meaning

• SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.87)
• Language Models Learn to Mislead Humans via RLHF (ICLR 2025 · similarity 0.86)
• Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.86)
• Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety (ICML 2025 · similarity 0.86)
• Fine-Tuning Language Models for Factuality (ICLR 2024 · similarity 0.85)
• Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis (ICLR 2024 · similarity 0.85)