Research map / Large language models / Alignment and preferences
Moral Alignment for LLM Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a small instruction-tuned LLM agent (Gemma2-2b-it) with PPO on intrinsic deontological and utilitarian moral rewards in the Iterated Prisoner's Dilemma, then tests unlearning of a selfish strategy and generalization to other matrix games.
Problem
Aligning LLM agents to human moral values without relying on opaque, implicit human preference data (RLHF/DPO), especially in agentic decision-making settings.
Approach
Replace human preference feedback with explicit intrinsic moral rewards, defined from deontological (do not defect against a cooperator) and utilitarian (collective payoff) frameworks, used in RL fine-tuning of an LLM agent playing social dilemma matrix games.
Key results
- Deontological fine-tuning makes the agent avoid defecting against a cooperator nearly 100% of the time during IPD training.
- Switching from Game to Deontological or Utilitarian reward at episode 500 partially unlearns the selfish strategy, though it does not reach the cooperation levels of purely prosocial training.
- Deontological-trained agents show very small moral regret when transferred zero-shot to four other matrix games with new action tokens; Utilitarian agents overly cooperate and do worse on coordination games.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with Deontological rewards that penalize defecting against a previous cooperator | Gemma2-2b-it · IPD vs TFT | avoid defecting against a cooperator: nearly 100% | Helped |
| Fine-tune with Utilitarian rewards based on collective payoff | Gemma2-2b-it · IPD vs TFT | mutual cooperation | Helped |
| Fine-tune with Game payoff rewards | Gemma2-2b-it · IPD vs TFT | defection | Helped |
| Fine-tune with Game+Deontological rewards | Gemma2-2b-it · IPD vs TFT | avoid defecting against a cooperator | Helped |
| Switch from Game rewards to Deontological or Utilitarian rewards during the second half of training | Game-trained Gemma2-2b-it · IPD | cooperation | Helped |
| Fine-tune on IPD moral rewards, then evaluate with new action tokens | Gemma2-2b-it · Iterated matrix games | Deontological moral regret | Helped |
| Fine-tune on Utilitarian rewards on IPD | Gemma2-2b-it · Iterated Defective Coordination | Utilitarian moral regret | Hurt |
| Fine-tune on IPD with intrinsic rewards | Gemma2-2b-it · Unrelated Action+Game prompts | responses using an action token: over 80% | Helped |
| Reverse which action token means Cooperate versus Defect during Game-reward training (ablation) | Gemma2-2b-it · IPD vs TFT | defection at training endpoint: nearly 100% | No clear effect |
| Replace action1/action2 with other tested action-token pairs (ablation) | Gemma2-2b-it fine-tuning · IPD training | training stability | Hurt |
| Use new action tokens at test time rather than the training tokens (ablation) | IPD-fine-tuned Gemma2-2b-it · Other matrix games | generalization of learned moral strategy | Helped |
| Swap the meanings of the original action tokens at test time (ablation) | IPD-fine-tuned Gemma2-2b-it · Other matrix games | moral regret | Hurt |
Surprising findings
- Fine-tuning on the IPD game also changed the model's responses to unrelated 'call to action' prompts without payoff matrices, suggesting spillover to general game scenarios.
- Swapping the meaning of training action tokens at test time changed behavior, indicating the model learned token semantics rather than abstract moral principles.
Limitations
- Requires explicitly specifying payoff-based rewards for each environment, unlike natural-language preference methods.
- Evaluated only on a single small model (Gemma2-2b-it) and simple matrix games with one-step history.
- Generalization is sensitive to action token choice and payoff matrix ordering, and the Utilitarian policy generalizes poorly to coordination games.
Experimental setup
Benchmarks: Iterated Prisoner's Dilemma Iterated Stag Hunt Iterated Chicken Iterated Bach or Stravinsky Iterated Defective Coordination
Metrics: moral regret (deontological) moral regret (utilitarian) action choice frequencies reciprocity
Baselines: Game (selfish) reward fine-tuning Tit-for-Tat
Builds on: PPO LoRA TRL
Closest papers by meaning
- MoralReason: Generalizable Moral Decision Alignment for LLM Agents Using Reasoning-Level Reinforcement Learning (AAAI 2026 · similarity 0.94)
- Many LLMs Are More Utilitarian Than One (NeurIPS 2025 · similarity 0.91)
- ME: Modelling Ethical Values for Value Alignment (AAAI 2025 · similarity 0.90)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.89)
- Learning Human-like Representations to Enable Learning Human Values (NeurIPS 2024 · similarity 0.89)
- SALMON: Self-Alignment with Instructable Reward Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.