Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Learning Human-like Representations to Enable Learning Human Values
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper shows that agents whose kernel (similarity-based representation) is more aligned with human similarity judgments learn human values faster, take fewer harmful actions during exploration, and generalize better, across simulations, morality judgments, and ten human values.
Problem
How to let AI systems learn individual human values quickly and safely during personalization, avoiding harmful actions while learning, and generalizing to unseen actions.
Approach
Treats representational alignment between human and model similarity judgments as a factor driving safe exploration and generalization when learning human values; uses fixed kernels from embedding models and LLM similarity ratings as agent representations in a multi-armed bandit, with a theoretical analysis predicting that misalignment in cross-kernel similarities (K*) hurts more than misalignment
Key results
- In simulations across three kernel agents (SVR, kernel regression, GPR; 2300+ experiments), higher representational alignment correlated with higher mean reward (e.g., rho=0.711 for kernel regression) and fewer immoral actions (rho=-0.749).
- Across 16 embedding and LLM similarity kernels on human morality judgments, alignment correlated with reward and bad actions; misalignment in cross-kernel similarities (K*) mattered more than training-kernel misalignment, as predicted by theory.
- Across 10 human values, most correlations were significant (p<0.0001), with generalization-phase correlations strongest (e.g., morality bad actions rho=-0.691 with full alignment); difficulty was an exception with no significant generalization effect.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase representational alignment with the simulated environment | Kernel-based agents · Simulated morality | Mean reward: 0.750; 0.711; 0.647 | Helped |
| Increase representational alignment with the simulated environment | Kernel-based agents · Simulated morality | Immoral actions taken: -0.798; -0.749; -0.696 | Helped |
| Increase representational alignment with the simulated environment | Kernel-based agents · Simulated morality | Iterations to convergence: -0.737; -0.721; -0.611 | Helped |
| Increase representational alignment with human judgments | Language-model kernel regression agent · Human morality, personalization | Mean reward: 0.36 | Helped |
| Increase representational alignment with human judgments | Language-model kernel regression agent · Human morality, generalization | Mean reward: 0.631 | Helped |
| Increase representational alignment with human judgments | Language-model kernel regression agent · Social Status, Compassion, Fairness, Honesty, Integrity, Loyalty, Popularity | Mean reward: 0.131; 0.326; 0.15; 0.075; 0.23; 0.35; 0.235 | Helped |
| Increase representational alignment with human judgments | Language-model kernel regression agent · Difficulty, generalization | Mean reward: -0.241 | No clear effect |
| Increase representational alignment with human judgments | Language-model kernel regression agent · Human values, personalization | Bad actions taken: 0.139; 0.132; 0.218; 0.095 | Mixed |
| Increase representational alignment with human judgments | Language-model kernel regression agent · Human values, generalization | Bad actions taken: -0.691; -0.662; -0.564; -0.34; -0.293; -0.506; -0.561; -0.577 | Helped |
| Increase representational alignment with human judgments | Text-based LLM agents · Morality, text-based RL | Mean reward: 0.1 | Helped |
| Increase representational alignment with human judgments | Text-based LLM agents · Morality, text-based RL | Immoral actions taken: -0.16 | Helped |
| Increase representational alignment with human judgments | Text-based LLM agents · Morality judgments | Squared difference from human morality scores: -0.149 | Helped |
Surprising findings
- Decreasing representational alignment beyond a certain point made kernel agents perform worse than the Thompson sampling baseline.
- LLM agents in text-based bandit settings showed a ceiling effect, with near-zero immoral actions, and 28 non-chatbot LLMs simply refused to respond.
Limitations
- Human value ratings came from English-speaking US internet users and are not representative of diverse cultural or individual perspectives.
- Restricted to a small set of 50 atomic action descriptions and a simple bandit setup; results may not extend to complex realistic environments or all architectures.
- Difficulty value showed no significant generalization effect, suggesting individualized values may not align with shared representations.
Experimental setup
Datasets: ETHICS (Justice category, adapted) human value and similarity judgments collected by authors
Benchmarks: simulated multi-armed bandit environments human morality judgments over 50 actions 10 human values (ethics, honesty, fairness, etc.)
Metrics: mean reward immoral/bad actions taken Spearman correlation with representational alignment non-optimal actions iterations to convergence
Baselines: Thompson sampling Doc2Vec Universal Sentence Encoder text-embedding-ada-002 GPT-4o GPT-4
Builds on: Thompson sampling Gaussian process regression kernel regression support vector regression ETHICS dataset
Closest papers by meaning
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.90)
- Deep Value Benchmark: Measuring Whether Models Generalize Deep values or Shallow Preferences (NeurIPS 2025 · similarity 0.89)
- Moral Alignment for LLM Agents (ICLR 2025 · similarity 0.89)
- MAP: Multi-Human-Value Alignment Palette (ICLR 2025 · similarity 0.89)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.89)
- Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.