Research map / Large language models / AI and society
ProgressGym: Alignment with a Millennium of Moral Progress
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces progress alignment and the ProgressGym framework, using 9 centuries of historical text and historical Llama3 LLMs to benchmark lifelong and extrapolative alignment algorithms on PG-Follow, PG-Predict, and PG-Coevolve challenges.
Problem
Current alignment methods use static human preferences and so risk locking in contemporary moral blindspots (value lock-in); there is no temporal framework or benchmark for aligning AI to moral progress.
Approach
Formulate progress alignment as a temporal POMDP over evolving human values, emulate historical human proxies with century-specific Llama3 models, and propose lifelong and extrapolative (finite-difference polynomial extrapolation of preference data) alignment baselines wrapping RLHF/DPO.
Key results
- Extrapolative 2,2 independent with RLHF achieves the top PG-Follow score of 6.753 and PG-Predict score of 29.489, while lifelong independent with RLHF achieves the top PG-Coevolve score of 38.645 (DPO extrapolative 2,2 reaches 38.959).
- First-order extrapolation is consistently outperformed by either lifelong following or second-order extrapolation.
- DPO outperforms RLHF in 13 of 15 back-to-back comparisons, per the authors' observations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use second-order extrapolation (2,2) instead of first-order extrapolation (1,1) | Extrapolative alignment pipeline · PG-Follow, PG-Predict and PG-Coevolve | score | Mixed |
| Use DPO rather than RLHF as the classical alignment algorithm | Progress alignment pipelines · ProgressGym benchmarks | score: 13 out of 15 | Helped |
| Use Extrapolative 2,2 instead of Lifelong | RLHF-based PG-Follow · PG-Follow | score | Helped |
| Choose the alignment pipeline based on the classical alignment algorithm | Progress alignment pipelines · PG-Follow, PG-Predict and PG-Coevolve | score | Mixed |
Surprising findings
- First-order extrapolation is consistently outperformed by both plain following and second-order extrapolation.
- Extrapolative second-order method beats lifelong on PG-Follow with RLHF despite being designed for prediction.
Limitations
- Limited cultural diversity in the historical text dataset (predominantly English).
- Historical proxy models show limited ability to reflect historical value trends.
- Results are exploratory; hypotheses are not rigorously tested and some algorithms fail (N/A) due to broken instruction following.
Experimental setup
Datasets: Internet Archive Project Gutenberg EEBO Pile of Law Alpaca LIMA Dolly-15k MoralChoice Moral Foundations Questionnaire Integrated Worldview Framework
Benchmarks: PG-Follow PG-Predict PG-Coevolve
Metrics: sum of cosine similarity between value embeddings (19 dimens
Baselines: Lifelong (iterative) Lifelong (independent) Extrapolative 1,1 Extrapolative 2,2
Builds on: RLHF DPO Dynamic Reward MDP On-the-fly Preference Optimization
Closest papers by meaning
- Language Model Alignment in Multilingual Trolley Problems (ICLR 2025 · similarity 0.88)
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.88)
- Position: A Roadmap to Pluralistic Alignment (ICML 2024 · similarity 0.88)
- Training-Free Cultural Alignment of Large Language Models via Persona Disagreement (NeurIPS 2026 · similarity 0.87)
- Moral Alignment for LLM Agents (ICLR 2025 · similarity 0.87)
- Unveiling the Bias Impact on Symmetric Moral Consistency of Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.