Research map / Multi-agent systems and games / LLM agents

MoralReason: Generalizable Moral Decision Alignment for LLM Agents Using Reasoning-Level Reinforcement Learning

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds Moral-Reason-QA (680 high-ambiguity MoralChoice scenarios with framework-specific reasoning traces for utilitarian, deontological, virtue ethics) and trains Qwen3-4B with GRPO plus a composite alignment/keyword reward to generalize moral frameworks to held-out scenarios.

Problem

Actively steering LLM agents' moral decisions toward a specified moral framework, and getting that behavior to generalize to scenarios outside the training distribution; prior work mostly evaluates morality or is limited to game-theoretic settings.

Approach

Cast framework-specific moral decision alignment as RL with GRPO, using a composite reward that combines a decision-alignment reward with a capped keyword reward on reasoning traces, and measure out-of-distribution alignment via a softmax-normalized alignment score.

Key results

• Utilitarian-aligned model raises OOD alignment score from 0.207 to 0.964 (+0.757) after GRPO.
• Deontological-aligned model raises score from 0.207 to 0.657 (+0.450), peaking at step 75.
• Virtue-aligned model shows no improvement; base model already biased toward virtue (0.586).

Surprising findings

• The base model's implicit virtue-ethics bias (0.586) made virtue-aligned GRPO training ineffective.
• Deontological performance declined after step 75, suggesting overfitting or instability.

Limitations

• Only three Western frameworks and a single base model (Qwen3-4B) tested.
• Virtue ethics alignment failed; deontological gains were unstable after step 75.
• Small evaluation set of 50 scenarios and keyword-based reasoning reward may not capture moral nuance.

Experimental setup

Datasets: MoralChoice

Benchmarks: Moral-Reason-QA held-out disagreement scenarios (50)

Metrics: softmax-normalized OOD alignment score alignment reward

Baselines: Qwen3-4B-Base before GRPO

Builds on: GRPO MoralChoice Qwen3-4B

Closest papers by meaning

• Moral Alignment for LLM Agents (ICLR 2025 · similarity 0.94)
• Many LLMs Are More Utilitarian Than One (NeurIPS 2025 · similarity 0.89)
• Human-Guided Moral Decision Making in Text-Based Games (AAAI 2024 · similarity 0.88)
• Language Model Alignment in Multilingual Trolley Problems (ICLR 2025 · similarity 0.88)
• Learning Human-like Representations to Enable Learning Human Values (NeurIPS 2024 · similarity 0.88)
• ME: Modelling Ethical Values for Value Alignment (AAAI 2025 · similarity 0.87)