Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Human-Guided Moral Decision Making in Text-Based Games
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HuMAL, a cycle of RL gameplay and human-agent collaboration in which sparse Likert-scale human ratings update a GPT-2 action generator and a RoBERTa moral prior, evaluated on Jiminy Cricket text games with simulated and real human feedback.
Problem
Text-based game RL agents need to achieve task goals while acting morally, but static moral priors from LMs generalize poorly because human morality is contextual and implicit, and step-by-step human feedback is impractical in large text-game action spaces.
Approach
Agents collect high-quality trajectories into a buffer, morally salient samples are selected by a moral prior and rated by humans on a Likert scale, and those ratings are used to supervise both the LM action generator (with a sign-weighted loss) and the moral prior, enabling personalized value learning with few feedback labels.
Key results
- On 10 Jiminy Cricket games with simulated feedback, HuMAL Policy Shaping reaches average Immorality Score 1.01 and Completion Percentage 6.75 versus CMPS 1.75 and 6.10, with RI 0.15 vs 0.29.
- HuMAL Policy Shaping's average Completion Percentage is 6.75, a 10.6% gain over CMPS, while immorality falls by 73%.
- In real human experiments on Zork1 with four participants, each agent was rated lower immorality by its own trainer (avg 1.07) than by others (avg 1.88), with completion around 9.76.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HuMAL policy shaping with limited human feedback instead of CMPS | Jiminy Cricket agents · Jiminy Cricket (10-game AVG) | Immorality Score: 1.75 -> 1.01 | Helped |
| Use HuMAL policy shaping with limited human feedback instead of CMPS | Jiminy Cricket agents · Jiminy Cricket (10-game AVG) | Completion Percentage: 6.10 -> 6.75 | Helped |
| Use HuMAL reward shaping instead of CMPS | Jiminy Cricket agents · Jiminy Cricket (10-game AVG) | Immorality Score: 1.75 -> 1.30 | Helped |
| Use HuMAL reward shaping instead of CMPS | Jiminy Cricket agents · Jiminy Cricket (10-game AVG) | Completion Percentage: 6.10 -> 6.63 | Helped |
| Add morality learning that improves the LM module (ablation) | HuMAL Policy Shaping · Jiminy Cricket (10-game AVG) | Completion Percentage: 5.90 -> 6.75 | Helped |
| Add morality learning that improves the LM module (ablation) | HuMAL Policy Shaping · Jiminy Cricket (10-game AVG) | Immorality Score: 0.98 -> 1.01 | No clear effect |
| Add morality learning that improves the moral prior (ablation) | HuMAL Policy Shaping · Jiminy Cricket (10-game AVG) | Immorality Score: 1.63 -> 1.01 | Helped |
| Add morality learning that improves the moral prior (ablation) | HuMAL Policy Shaping · Jiminy Cricket (10-game AVG) | Completion Percentage: 7.03 -> 6.75 | Hurt |
| Improve both the LM module and moral prior during morality learning (ablation) | HuMAL Policy Shaping · Jiminy Cricket (10-game AVG) | Immorality Score: 1.75 -> 1.01 | Helped |
| Improve both the LM module and moral prior during morality learning (ablation) | HuMAL Policy Shaping · Jiminy Cricket (10-game AVG) | Completion Percentage: 6.10 -> 6.75 | Helped |
Surprising findings
- Participants consistently rated their own personalized agent as less immoral than agents trained by other participants, showing value-specific adaptation.
Limitations
- Real human experiments use only four participants and a single game (Zork1).
- Simulated experiments rely on pre-defined Jiminy Cricket annotations rather than real feedback; the moral prior depends on out-of-distribution performance.
- Only actions targeted at others are considered in the immorality measure.
Experimental setup
Datasets: Jiminy Cricket ETHICS commonsense morality ClubFloyd
Benchmarks: Jiminy Cricket
Metrics: Immorality Score Completion Percentage Relative Immorality
Baselines: CMPS CMRS CMPS (Oracle) HuMAL Reward Shaping
Builds on: CALM DRRN CMPS CMRS Jiminy Cricket
Closest papers by meaning
- Moral Alignment for LLM Agents (ICLR 2025 · similarity 0.88)
- MoralReason: Generalizable Moral Decision Alignment for LLM Agents Using Reasoning-Level Reinforcement Learning (AAAI 2026 · similarity 0.88)
- Learning Human-like Representations to Enable Learning Human Values (NeurIPS 2024 · similarity 0.87)
- Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gain (ICLR 2024 · similarity 0.84)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.84)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.