Research map / Reinforcement learning / Policy gradient and actor-critic

FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Tree search and policy optimization · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FairDICE, an offline MORL method that directly optimizes Nash social welfare (alpha-fairness) via DICE-style stationary distribution correction, learning implicit preference weights without preference conditioning or weight search.

Problem

Fair multi-objective RL (optimizing nonlinear welfare such as Nash social welfare or max-min fairness) in the offline setting from a fixed dataset, where linear scalarization cannot capture fairness trade-offs.

Approach

Formulates regularized welfare optimization over stationary distributions with an f-divergence to the data distribution, derives a sample-based dual loss (OptiDICE extension) where per-objective multipliers mu act as implicit preference weights learned jointly with the policy, and extracts the policy via weighted behavior cloning.

Key results

• On MO-Four-Room, FairDICE-NSW achieves NSW of -10.77 versus -17.33 for utilitarian and -16.10 for behavior, with Jain's index 0.996.
• FairDICE achieves competitive or superior NSW across all five two-objective D4MORL tasks compared with best-of-weight-search preference-conditioned baselines (BC(P), MODT(P), MORvS(P)).
• On MO-Hopper-3obj, FairDICE finds a welfare-maximizing policy without preference conditioning, with raw returns close to or beyond the Pareto frontier of baselines.

Measured effects reported in the paper

Surprising findings

• Higher alpha (more fairness) sustains strong utilitarian performance across a wide range of beta, due to concavity tempering return maximization and implicitly regularizing distribution shift.
• Preference-conditioned baselines with exhaustive weight search are outperformed by FairDICE even though FairDICE searches no weights.

Limitations

• Only handles strictly concave (convex-formulation) scalarization; non-convex nonlinear scalarizations are not covered.
• Performance depends on hyperparameters controlling distribution shift (beta) and dataset quality.
• Experiments are limited to small finite toy domains and D4MORL two- and three-objective tasks.

Experimental setup

Datasets: D4MORL D4RL

Benchmarks: D4MORL MO-Hopper MO-Walker2d MO-HalfCheetah MO-Ant MO-Swimmer MO-Hopper-3obj MO-Four-Room Random MOMDP

Metrics: Nash social welfare Utilitarian welfare Jain's fairness index raw returns

Baselines: BC(P) MODT(P) MORvS(P) FairDICE-fixed

Builds on: OptiDICE DICE-RL D4RL D4MORL Eisenberg-Gale program

Closest papers by meaning

• Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.90)
• AETDICE: Unified Framework and Offline Optimization for Nonlinear Multi-Objective RL (NeurIPS 2026 · similarity 0.90)
• Intersectional Fairness in Reinforcement Learning with Large State and Constraint Spaces (ICML 2025 · similarity 0.88)
• MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning (ICML 2025 · similarity 0.88)
• On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.87)
• ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization (ICLR 2025 · similarity 0.87)