Research map / Reinforcement learning / Policy gradient and actor-critic

AETDICE: Unified Framework and Offline Optimization for Nonlinear Multi-Objective RL

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Tree search and policy optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the AET framework unifying SER and ESR nonlinear MORL via trajectory-level transformation F and concave expectation-level aggregation G, and AETDICE, an offline DICE-style algorithm operating in an augmented state space.

Problem

Offline optimization of nonlinear multi-objective RL objectives: ESR (non-Markovian, trajectory-level nonlinearity) and SER (global, expectation-level nonlinearity) lack a unified offline method, and their combination (AET) is unaddressed.

Approach

Decompose nonlinear scalarization into trajectory-level transformation F, expectation, and concave aggregation G; absorb F into a transformed per-step reward in an augmented state (state, accumulated reward, time), then solve a finite-horizon DICE dual with a slack variable to handle concave G.

Key results

• On MO-Ant with convex F and linear G, ESR-IQL reaches 1251.46 ± 17.88 versus AETDICE 953.53 ± 16.31, while with convex F and concave G AETDICE reaches 13.69 ± 0.05, the best among methods (FairDICE 13.04 ± 0.07).
• In Fair-Taxi, ESR-optimal policies consistently serve both groups within an episode, whereas SER-optimal policies specialize per trajectory and balance only in expectation.
• SER-optimal policies show higher expected returns but larger variance across trajectories than ESR-optimal policies in MO-PointMaze-3obj.

Surprising findings

• On MO-Ant with linear G, ESR-IQL outperforms AETDICE, the full DICE method, in per-objective return, indicating that the augmented-space DICE method is not uniformly best.
• Convex F with linear G causes the policy to specialize in one direction even under equal weighting, unlike linear MORL.

Limitations

• Coverage assumption over accumulated reward in augmented space is stronger than standard DICE coverage and may be hard to satisfy.
• Augmented state space is continuous and high-dimensional, making DICE in augmented space harder than the original MOMDP.
• Cobb-Douglas utility with low rho can cause agent collapse without an additional safety utility.

Experimental setup

Datasets: D4MORL Fair-Taxi MO-PointMaze

Benchmarks: Fair-Taxi MO-PointMaze-3obj MO-Ant MO-Walker2d MO-Halfcheetah

Metrics: expected return per-trajectory return quantiles AET objective value

Baselines: OptiDICE FairDICE ESR-IQL Linear MORL with fixed weights

Builds on: OptiDICE FairDICE IQL DICE-RL Nonlinear MORL ESR augmented state formulation

Closest papers by meaning

• FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.89)
• Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF (ICML 2025 · similarity 0.88)
• Pareto Set Learning for Multi-Objective Reinforcement Learning (AAAI 2025 · similarity 0.87)
• On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.87)
• MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning (ICML 2025 · similarity 0.87)