Research map / Large language models / Alignment and preferences
Deep Value Benchmark: Measuring Whether Models Generalize Deep values or Shallow Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Deep Value Benchmark, an in-context confound-then-deconfound test that measures whether 9 LLMs generalize deep moral values or shallow preferences (DVGR), finding an average DVGR of 0.30.
Problem
Whether LLMs learn fundamental human values or merely surface-level preferences from preference data, which matters for alignment because shallow-correlation learning can produce misaligned behavior in novel contexts.
Approach
Deliberately confound deep values (prima facie duties and Schwartz values) with shallow preferences during in-context training, then decouple them at test time and measure the rate at which models pick the value-aligned option (DVGR).
Key results
- Average DVGR across 9 models is 0.30; every model scores significantly below chance (0.5).
- Larger models have slightly lower DVGR than smaller counterparts (smaller models higher in 3/5 pairs); scale does not help.
- Explicit instruction to generalize deep values raises pooled DVGR to 0.33, while chain-of-thought lowers it to 0.25.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Explicitly instruct models to generalize the deep value rather than the shallow preference (ablation) | Models tested on DVB · DVB | DVGR: 0.30 -> 0.33 | Helped |
| Add Chain-of-Thought reasoning without explicit guidance (ablation) | Models tested on DVB · DVB | DVGR: 0.30 -> 0.25 | Hurt |
| Increase the number of in-context examples from 4 to 20 or 40 (ablation) | Models tested on DVB · DVB | DVGR: 0.31 -> 0.30; 0.31 -> 0.30 | No clear effect |
| Use larger rather than smaller models | Paired models from the same families · DVB | DVGR: mean absolute DVGR difference = 0.07 | Hurt |
| Generalize from preference examples where deep values and shallow features are confounded | Nine tested models · DVB | DVGR: 0.30 | Hurt |
Surprising findings
- Chain-of-thought reasoning decreased DVGR, with rationales amplifying shallow features.
- Larger models generalized deep values slightly less than smaller models within the same families.
Limitations
- Perfect artificial confound between deep values and shallow preferences may not reflect natural co-occurrence.
- Deep value generalization may not always be the appropriate or expected choice; DVGR is a tendency measure rather than a strict target.
- Only in-context learning tested; no task-specific fine-tuning evaluated.
Experimental setup
Datasets: LLM-generated shallow preference dichotomies Y Combinator AI assistant startup tags O*NET work activities
Benchmarks: Deep Value Benchmark
Metrics: Deep Value Generalization Rate (DVGR) accuracy Cramer's V chi-squared test
Baselines: chain-of-thought prompting explicit instruction prompting
Builds on: Schwartz's theory of basic values W.D. Ross prima facie duties chain-of-thought
Closest papers by meaning
- Strong Preferences Affect the Robustness of Preference Models and Value Alignment (ICLR 2025 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.91)
- Can DPO Learn Diverse Human Values? A Theoretical Scaling Law (NeurIPS 2025 · similarity 0.90)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.90)
- Learning Human-like Representations to Enable Learning Human Values (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.