Research map / Large language models / AI and society
Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Generative Psychometrics for Values (GPV), an LLM-based paradigm that parses blog text into value-laden perceptions, scores them with a fine-tuned Llama-3-8B (ValueLlama), and aggregates them to measure human and LLM values.
Problem
Existing value measurement tools (self-report questionnaires and dictionary-based methods) are static, inflexible, prone to response bias, and fail to capture nuanced semantics, both for humans and for LLMs.
Approach
Dynamically parse unstructured texts into perceptions (analogous to psychometric items), measure their relevance and valence to open-vocabulary values with a fine-tuned LLM, and aggregate to individual or LLM-level value profiles, applying the same paradigm to LLMs' free-form outputs.
Key results
- ValueLlama reaches 90.0% relevance and 91.5% valence accuracy on held-out ValueBench items, beating GPT-4 Turbo (79.8%/87.5%) and Kaleido (83.5%/82.5%).
- GPV-based linear probing predicts LLM safety scores with 85.6% accuracy vs 67.8% for ValueBench and 56.7% for self-report.
- GPV produces construct-valid correlations for LLM values (avg 0.57 on theoretically positive pairs) while self-report and ValueBench give 0.21 and -0.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPV rather than self-report or ValueBench to measure LLM values | LLM safety prediction with linear probing · LLM safety prediction (17 LLMs) | classification accuracy: 56.7 ± 26.0; 67.8 ± 20.6 -> 85.6 ± 14.1 | Helped |
| Use GPV rather than Self-report or ValueBench | LLM value measurement · Correlations between theoretically positively related values | average correlation: 0.21; -0.01 -> 0.57 | Helped |
| Use ValueLlama rather than Kaleido or GPT-4 Turbo for relevance classification | Perception-level value measurement · ValueBench held-out perception classification | relevance accuracy: 83.5%; 79.8% -> 90.0% | Helped |
| Use ValueLlama rather than Kaleido or GPT-4 Turbo for valence classification | Perception-level value measurement · ValueBench held-out perception classification | valence accuracy: 82.5%; 87.5% -> 91.5% | Helped |
| Use VSM rather than Schwartz as the value system | GPV safety prediction · LLM safety prediction (17 LLMs) | classification accuracy: 73 -> 86 | Helped |
| Use VSM rather than NFCC as the value system | GPV safety prediction · LLM safety prediction (17 LLMs) | classification accuracy: 57 -> 86 | Helped |
| Use VSM rather than LVI as the value system | GPV safety prediction · LLM safety prediction (17 LLMs) | classification accuracy: 72 -> 86 | Helped |
| Use VSM features rather than VSM combined with other systems | GPV safety prediction · LLM safety prediction (17 LLMs) | classification accuracy: 86; 72; 76; 67; 75; 67; 75; 73 | Mixed |
| Long Term Orientation contributes positively to the safety prediction | VSM-based GPV safety prediction · LLM safety prediction (17 LLMs) | linear-probe parameter: 0.80 | Helped |
| Masculinity contributes negatively to the safety prediction | VSM-based GPV safety prediction · LLM safety prediction (17 LLMs) | linear-probe parameter: -0.46 | Hurt |
| Aggregate perception-level measurements into individual-level values | GPV for human blogs · Blog Authorship Corpus | perception-level and individual-level agreement: 86.6% | Helped |
| Aggregate perception-level measurements into model-level values | GPV for LLMs · LLM value measurements | perception-level and LLM-level agreement: 86.8% | Helped |
Surprising findings
- Prior self-report and ValueBench measurements of LLMs show almost all-positive value correlations, contrary to theoretical structure, indicating response bias.
- Schwartz's value system is less predictive of LLM safety than the VSM value system (86% vs lower for Schwartz-based combinations).
Limitations
- Evaluation limited to English-language texts and LLMs.
- Blog data mainly covers everyday topics and rarely controversial issues; parsing quality validated by annotators with similar demographic backgrounds.
- Relies on GPT-3.5 Turbo for perception parsing and LLM-generated value-eliciting questions, inheriting LLM biases.
Experimental setup
Datasets: ValueBench ValuePrism Blog Authorship Corpus SALAD-Bench Salad-Data
Benchmarks: ValueBench held-out items Blog Authorship Corpus SALAD-Bench
Metrics: accuracy Pearson correlation stability agreement rate linear probing accuracy
Baselines: Self-report questionnaires ValueBench Personal Values Dictionary (PVD) Kaleido GPT-4 Turbo
Builds on: ValueBench ValuePrism Personal Values Dictionary SALAD-Bench
Closest papers by meaning
- CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses (NeurIPS 2024 · similarity 0.89)
- Deep Value Benchmark: Measuring Whether Models Generalize Deep values or Shallow Preferences (NeurIPS 2025 · similarity 0.89)
- Do LLMs have Consistent Values? (ICLR 2025 · similarity 0.89)
- Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets (NeurIPS 2024 · similarity 0.87)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.87)
- ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.