Research map / Large language models / AI and society
An Evolutionary Perspective on AI Alignment (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Models RLHF as a two-player game (aligned vs misaligned strategies) under replicator dynamics and shows a biased human judge makes training converge to misalignment.
Problem
Understanding whether RLHF value elicitation from human judges reliably produces aligned behavior, especially when judge preferences are biased, where quantitative benchmarks are hard to define.
Approach
Applies evolutionary game theory (replicator dynamics) to a parametrized 2x2 game of RLHF, where payoffs encode the human judge's preferences; analysis of rest points and stability shows that a neutral judge yields a stable aligned equilibrium while a biased judge yields a misaligned dominance outcome.
Key results
- Under a neutral judge (a, b, c, d ≈ 0.5), the cooperative rest point (A, A) is the stable attractor, supporting RLHF for alignment.
- When the judge is biased (a, c < ϵ or b, d < ϵ), the interior rest point becomes unstable and the globally stable outcome is (MA, A) or (A, MA), so RLHF trains for misalignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a judge without strong bias toward either model | RLHF training under replicator dynamics · RLHF game | aligned behavior | Helped |
| use a judge biased in favor of one model | RLHF training under replicator dynamics · RLHF game | aligned behavior | Hurt |
Surprising findings
- A biased judge makes RLHF incentivize misalignment in the model population rather than alignment.
Limitations
- Analysis is restricted to a simple parametrized 2x2 game with infinite populations and no iteration or population-size bounds.
- Empirical validation with actual LLM training is not provided.
Experimental setup
Builds on: RLHF replicator dynamics Safety-by-Debate Cooperative Inverse Reinforcement Learning
Closest papers by meaning
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.91)
- Moral Alignment for LLM Agents (ICLR 2025 · similarity 0.89)
- Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment (AAAI 2025 · similarity 0.89)
- AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.89)
- Learning Human-like Representations to Enable Learning Human Values (NeurIPS 2024 · similarity 0.89)
- Improving Generalization of Alignment with Human Preferences through Group Invariant Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.