Research map / Reinforcement learning / Policy gradient and actor-critic

Reward Dimension Reduction for Scalable Multi-Objective Reinforcement Learning

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an online reward dimension reduction method using a row-stochastic, positive affine map learned with a reconstruction network, so multi-policy MORL scales to many objectives while preserving Pareto-optimality.

Problem

Multi-policy multi-objective RL becomes hard to scale as the number of objectives grows (e.g., 16), and standard dimension reduction methods (PCA, autoencoders, NPCA) are batch-oriented and fail to preserve Pareto-optimality when applied online.

Approach

Reduce the vector reward online with an affine map A whose entries are positive and whose rows sum to one (softmax-parameterized), so that preference vectors in the reduced simplex map to valid preferences in the original simplex and Pareto-optimality is preserved; A is trained jointly with a reconstruction network from the reduced reward back to the original reward.

Key results

• In the 16-objective traffic environment, the proposed method improves hypervolume over the base algorithm by a factor of 37.9 (166.9 vs 4.4 x10^61) and reduces sparsity to 2.3 x10^5 from 1842 x10^5.
• In LunarLander-5D, hypervolume is 25.6 x10^7 vs 3.1 x10^7 for the base algorithm, with sparsity 1.1 x10^2 vs 31.2.
• Removing positivity of the reduction matrix yields zero hypervolume, indicating positivity is required for Pareto-optimality preservation in the ablation.

Measured effects reported in the paper

Surprising findings

• Online PCA and autoencoder reductions perform worse than the base algorithm without reduction, and in the traffic task PCA and AE yield near-zero hypervolume.
• Strict monotone neural reward mappings without the affine positive constraint produce zero hypervolume.

Limitations

• Positivity and row-stochasticity are only sufficient conditions for Pareto-optimality; necessary conditions are not established.
• Only two environments were tested, and no established benchmark exists with more than ten objectives.
• Test-time preferences in the original high-dimensional space must still be mapped to the reduced space, which is not solved.

Experimental setup

Datasets: LunarLander-5D sumo-rl traffic light control (modified to 16 objectives)

Benchmarks: LunarLander-5D traffic light control with 16 objectives

Metrics: hypervolume sparsity expected utility metric (EUM)

Baselines: Base (MORL algorithm without reduction) incremental PCA online autoencoder online NPCA NPCA-ortho Monotone reward mapping

Builds on: morl-baselines non-negative PCA incremental PCA online autoencoder multi-objective Q-learning with optimality operator

Closest papers by meaning

• The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm (ICML 2024 · similarity 0.88)
• Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach (NeurIPS 2025 · similarity 0.88)
• On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.88)
• Pareto Set Learning for Multi-Objective Reinforcement Learning (AAAI 2025 · similarity 0.87)
• Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.87)
• Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF (ICML 2025 · similarity 0.87)