Research map / Large language models / Alignment and preferences
Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Value Imprint, which builds a human-values taxonomy from philosophy/ethics literature, annotates 6,501 Anthropic hh-rlhf preferences, trains a RoBERTa classifier on those labels, and audits three RLHF datasets for embedded values.
Problem
Little is known about which specific human values are operationalized in RLHF preference datasets, and no method existed to audit and measure them.
Approach
A taxonomy of seven human-value categories derived from an integrated literature review in philosophy, axiology, and STS, used as a codebook to annotate RLHF preferences and train a RoBERTa classifier that audits full datasets at scale.
Key results
- Human evaluation of 500 classification results showed the model predicted the correct value 84% of the time.
- Information-utility values (Wisdom/Knowledge and Information Seeking) were the most common across all three datasets, e.g., Wisdom/Knowledge at 78.17% in OpenAI WebGPT and 66.56% in Alpaca GPT-4.
- Justice & Human/Animal Rights was least represented (0.04% in WebGPT, 0.17% in Alpaca GPT-4, 1.76% in Anthropic hh-rlhf chosen); inter-annotator Krippendorff's alpha was 0.85.
Surprising findings
- Some preferences selected as suitable in the chosen column contained unethical or anti-social content, such as an example demeaning women.
Limitations
- Taxonomy and labels are primarily Western-focused and may not transfer to non-Western preferences.
- Values are subjectively assigned; the classifier struggles on underrepresented categories such as Empathy & Helpfulness (F1 0.629) and Well-being & Peace (F1 0.649).
- Taxonomy does not cover code or math RLHF.
Experimental setup
Datasets: Anthropic/hh-rlhf OpenAI WebGPT Comparisons Alpaca GPT-4-LLM
Benchmarks: Anthropic/hh-rlhf annotated subset (held-out 20%)
Metrics: accuracy F1-score Krippendorff's alpha
Builds on: RoBERTa Schwartz's theory of basic human values
Closest papers by meaning
- Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models (AAAI 2025 · similarity 0.87)
- Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties (AAAI 2024 · similarity 0.87)
- DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life (ICLR 2025 · similarity 0.86)
- CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses (NeurIPS 2024 · similarity 0.85)
- Deep Value Benchmark: Measuring Whether Models Generalize Deep values or Shallow Preferences (NeurIPS 2025 · similarity 0.85)
- More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.