Research map / Reinforcement learning / Policy gradient and actor-critic
ME: Modelling Ethical Values for Value Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes "modelled-ethics" reward functions for RL value alignment, grounding rewards in established domain-specific mathematical models instead of hand-tuned values, and tests them in three 20x20 gridworld UAV scenarios (disaster, ecocentrism, healthcare).
Problem
Hand-programmed, manually tuned case-based reward functions for RL-based value alignment are inconsistent and depend on the programmer's arbitrary numeric choices and on the environment.
Approach
Replace hand-set numeric penalties with rewards computed from existing domain-specific mathematical models (disaster damage regression, Lotka-Volterra predator-prey dynamics, power-law survival rate), used with standard tabular Q-learning.
Key results
- Modelled-ethics agent gave higher total damage relieved than the average hand-programmed agent in 4/5 gridworlds in the disaster case.
- In ecology, the modelled-ethics agent caused zero wildlife population and dynamic change across all 5 gridworlds, with hand-programmed agents significantly worse (Kruskal-Wallis p < 0.05).
- In healthcare, the modelled-ethics agent had higher expected survival than the average hand-programmed agent, but never exceeded the best hand-programmed agent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the modelled-ethics reward function instead of hand-programmed rewards | Q-learning agent · Disaster gridworlds | total damage relieved: 4/5 gridworlds | Mixed |
| Use the modelled-ethics reward function instead of hand-programmed rewards | Q-learning agent · Disaster gridworlds | average damage relieved per state: 3/5 gridworlds | Mixed |
| Use the modelled-ethics reward function instead of hand-programmed rewards | Q-learning agent · Ecology gridworlds | wildlife population change: 0; Kruskal-Wallis p-value < 0.05 | Helped |
| Use the modelled-ethics reward function instead of hand-programmed rewards | Q-learning agent · Ecology gridworlds | wildlife dynamic change: 0; Kruskal-Wallis p-value < 0.05 | Helped |
| Use the modelled-ethics reward function instead of hand-programmed rewards | Q-learning agent · Healthcare gridworlds | expected survival rate | Helped |
Surprising findings
- The best hand-programmed reward iteration sometimes outperformed the modelled-ethics agent in the disaster and healthcare cases.
- A simple hand-programmed reward (punishing every state spent) was as effective as the model-based reward in the simple healthcare task.
Limitations
- Experiments are limited to simple, small 20x20 gridworlds rather than real-world applications.
- Single programmer performed hand-tuning, which may introduce bias; approach requires careful model selection and terminal-state value setting.
- Modelled approach has limited generalization across diverse scenarios.
Experimental setup
Datasets: Bennitt et al. (2019) terrestrial mammal responses to UAVs Song and Park (2019) disaster damage formulas
Benchmarks: 5 custom 20x20 gridworlds per ethical scenario (disaster, ecology, healthcare)
Metrics: total damage relieved average damage relieved per state wildlife population change wildlife dynamic change expected survival rate of patient
Baselines: hand-programmed reward functions
Builds on: Q-learning Case-based reward functions Song and Park (2019) disaster damage prediction Lotka-Volterra predator-prey model Pourghaderi et al. (2022) survival rate model
Closest papers by meaning
- Moral Alignment for LLM Agents (ICLR 2025 · similarity 0.90)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.87)
- Learning Human-like Representations to Enable Learning Human Values (NeurIPS 2024 · similarity 0.87)
- MoralReason: Generalizable Moral Decision Alignment for LLM Agents Using Reasoning-Level Reinforcement Learning (AAAI 2026 · similarity 0.87)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.86)
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.