Research map / Reinforcement learning / Offline RL

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning

ICML 2025 · Reinforcement learning › Offline RL · approach: Diffusion models for offline decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MODULI, a preference-conditioned DiT diffusion planner with two multi-objective return normalization schemes and a slider adapter that shifts generation from in-distribution to out-of-distribution preferences for offline MORL.

Problem

Offline multi-objective RL policies generalize poorly to out-of-distribution preferences when offline datasets cover preferences narrowly or with gaps, yielding incomplete Pareto fronts.

Approach

Casts offline MORL as preference- and return-conditioned diffusion planning with classifier-free guidance, adds Preference Predicted and Neighborhood Preference return normalization, and trains a slider adapter that learns the direction of preference change so the sampled trajectory distribution can be shifted to unseen preferences at inference.

Key results

• On D4MORL Complete datasets, MODULI achieves the highest hypervolume in most Amateur and Expert environments (e.g., Hopper-Amateur HV 2.01×10^7 vs. MORvS 1.76×10^7) with the lowest sparsity in most tasks.
• On Shattered datasets, MODULI reduces return deviation substantially versus MORvS(P) (e.g., Hopper RD 0.22×10^3 vs. 1.10×10^3 and Walker2d RD 0.15×10^3 vs. 0.65×10^3).
• Neighborhood Preference Normalization gives the highest HV among the three normalization schemes on the Complete Amateur Hopper and Walker2d datasets (1.98×10^7 and 5.03×10^6).

Measured effects reported in the paper

Surprising findings

• Global normalization yields the sparsest (lowest SP) fronts on some amateur datasets yet much lower hypervolume, indicating dense but low-quality solutions.
• Preference Predicted Normalization underperforms on low-quality amateur datasets because the return predictor overestimates the achievable maximum.

Limitations

• Experiments are limited to low-dimensional state spaces and continuous action spaces; no image inputs.
• Nonlinear preference spaces are not addressed and may pose new challenges.
• Slider-based OOD generalization is evaluated only on D4MORL MuJoCo tasks and uses a linear preference space.

Experimental setup

Datasets: D4MORL Shattered (derived from D4MORL Complete) Narrow (derived from D4MORL Complete)

Benchmarks: D4MORL

Metrics: hypervolume sparsity return deviation

Baselines: MODT(P) MORvS(P) BC(P) MOCQL(P)

Builds on: Diffusion Forcing DDIM classifier-free guidance D4MORL MORvS DiT

Closest papers by meaning

• Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.91)
• On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.90)
• FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Forward KL Regularized Preference Optimization for Aligning Diffusion Policies (AAAI 2025 · similarity 0.88)
• Preference Elicitation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
• Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)