Research map / Reinforcement learning / Visual and representation RL

On Generalization Across Environments In Multi-Objective Reinforcement Learning

ICLR 2025 · Reinforcement learning › Visual and representation RL · approach: Probabilistic and autonomous RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalizes generalization in multi-objective RL via MOC-MDPs, introduces the MORL-Generalization benchmark with parameterized domains and the NHGR metric, and shows SOTA MORL algorithms generalize poorly.

Problem

Generalization across diverse environments in multi-objective reinforcement learning (MORL) has been unstudied, while RL generalization work assumes scalar rewards; evaluating MORL generalization is also non-trivial because the output is an approximate Pareto front per environment.

Approach

Extends the contextual MDP framework to multi-objective settings (MOC-MDP), defines axiomatic and utility-based generalization objectives, and proposes the Normalized Hypervolume Generalization Ratio (NHGR) to score generalist agents against per-context optimal Pareto fronts, evaluated on a parameterized benchmark with domain randomization.

Key results

• GPI-LS achieved an IQM NHGR of 0.66 on MO-LunarLander, with optimality gap 33.9%.
• On MO-HalfCheetah and MO-Humanoid, top algorithms reached IQM NHGR of only 0.28 and 0.41.
• On MO-SuperMarioBros the leading algorithm GPI-LS reached IQM NHGR of only 0.07 (optimality gap 87.7%).

Measured effects reported in the paper

Surprising findings

• MORL algorithms outperformed a single-objective SAC agent on its own scalar reward across MO-Hopper test environments during generalization training.
• Zero-shot generalization to held-out Mario stage 3-3 yielded negligible NHGR, suggesting memorization rather than generalization.

Limitations

• Gymnasium CPU-based rollouts create computational bottlenecks, making exhaustive hyperparameter tuning impractical.
• Benchmark focuses on state-space and dynamics variations; observation and reward variations are not emphasized.
• Baseline evaluations use no tailored generalization interventions and rely on hyperparameters adapted from single-environment settings.

Experimental setup

Datasets: MO-LunarLander MO-Hopper MO-HalfCheetah MO-Humanoid MO-SuperMarioBros MO-LavaGrid

Benchmarks: MORL-Generalization (MO-LunarLander, MO-Hopper, MO-HalfCheetah, MO-Humanoid, MO-SuperMarioBros, MO-LavaGrid)

Metrics: NHGR EUGR hypervolume expected utility metric normalized hypervolume IQM optimality gap

Baselines: CAPQL Envelope GPI-LS PCN PGMORL MORL/D SB GPI-PD MORL/D SB+PSA SAC

Builds on: Contextual MDP MO-Gymnasium GPI-LS MORL/D rliable

Closest papers by meaning

• Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.91)
• An Analytical Study of Utility Functions in Multi-Objective Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
• Pareto Set Learning for Multi-Objective Reinforcement Learning (AAAI 2025 · similarity 0.90)
• MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning (ICML 2025 · similarity 0.90)
• Multi-objective Linear Reinforcement Learning with Lexicographic Rewards (ICML 2025 · similarity 0.89)
• Horizon Generalization in Reinforcement Learning (ICLR 2025 · similarity 0.89)