Research map / Multi-agent systems and games / Games and equilibria
Moral Uncertainty and the Problem of Fanaticism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalises the problem of fanaticism in moral uncertainty as a property of social welfare functionals and proposes weighted k-trimmed Highest Mean and Highest Median as non-fanatical alternatives to Maximising Expected Choiceworthiness.
Problem
Maximising Expected Choiceworthiness (MEC) for moral uncertainty lets low-credence ethical theories with extreme evaluations dominate decisions (fanaticism), and fanaticism had never been mathematically defined.
Approach
Model moral uncertainty as social welfare aggregation (theories as individuals, credences as weights), define graded k-fanaticism via dominant subsets of low-credence theories, and propose weighted trimmed-mean and weighted-median aggregators that are provably less or non-fanatical than MEC.
Key results
- MEC and Maximin are proven Pascalian (fanatical for every credence level k in (0,0.5)).
- k-trimmed Highest Mean is not k'-fanatical for any k' <= k but is k*-fanatical for any k* > k.
- Highest Median is not k-fanatical for any k in (0,0.5), i.e. completely non-fanatical.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MEC for moral evaluation aggregation | MEC · Arbitrary ethical frameworks and action sets | fanaticism | Hurt |
| use Maximin for moral evaluation aggregation | Maximin · Arbitrary ethical frameworks and action sets | fanaticism | Hurt |
| trim the highest and lowest evaluations by credence mass k | k-trimmed Highest Mean · Arbitrary ethical frameworks and action sets | fanaticism | Helped |
| aggregate evaluations using the weighted median instead of the mean | Highest Median · Arbitrary ethical frameworks and action sets | fanaticism | Helped |
Surprising findings
- Highest Median ignores all other theories whenever a single theory has more than 0.5 credence, regardless of stakes.
- Maximin is fanatical through pessimistic low-credence theories, not only high-stakes ones.
Limitations
- Results are purely theoretical with a single running example; no empirical evaluation.
- Assumes ratio-scale full interpersonal comparability (RFC) of theory evaluations.
- Highest Median disregards stakes entirely when one theory has credence above 0.5; optimal k for trimmed mean left as future work.
Experimental setup
Metrics: k-fanaticism (graded) Pascalian fanaticism
Baselines: MEC Maximin
Builds on: maximising expected choiceworthiness social welfare aggregation majority judgment
Closest papers by meaning
- Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties (AAAI 2024 · similarity 0.80)
- DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life (ICLR 2025 · similarity 0.79)
- Risk-Sensitive Theory of Mind: Coordinating with Agents of Unknown Bias using Cumulative Prospect Theory (ICML 2025 · similarity 0.78)
- Operationalising Rawlsian Ethics for Fairness in Norm Learning Agents (AAAI 2025 · similarity 0.78)
- Many LLMs Are More Utilitarian Than One (NeurIPS 2025 · similarity 0.78)
- Weak Strategyproofness in Randomized Social Choice (AAAI 2025 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.