Research map / Optimization / Optimizers
Aligned Multi Objective Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Pearl, a modular, PyTorch-based production-ready reinforcement learning agent library with exploration, safety, history summarization, offline RL and contextual bandit support, with preliminary benchmarks and industry adoption examples.
Problem
Existing open-source RL libraries neglect production-critical needs such as exploration, safety/risk constraints, partial observability, offline RL, dynamic action spaces, and unified bandit/MDP support.
Approach
A modular RL agent library where policy learners, exploration modules, replay buffers, safety modules and history summarization modules can be independently mixed and matched, unifying bandit and MDP settings in one PyTorch framework.
Key results
- LSTM history summarization lets DQN improve on partially observable CartPole, where plain DQN made no learning progress, though neither reached the optimal return of 200.
- Bootstrapped DQN significantly outperformed DQN on small (5x5) and large (10x10) DeepSea exploration tasks.
- QR-DQN chose the lower-variance action when beta was high, consistent with the predicted 0.25 threshold in the stochastic bandit sanity check.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LSTM history summarization (ablation) | DQN · Partial-obobservable CartPole | return | Helped |
| Remove history summarization (ablation) | DQN · Partial-obobservable CartPole | return | Hurt |
| Use Bootstrapped DQN instead of DQN | Pearl agent · DeepSea (n=5 and n=10) | return | Helped |
| Increase β to favor lower return variance (ablation) | QR-DQN · Stochastic Bandit | greedy action | Helped |
| Decrease β to favor expected return over variance (ablation) | QR-DQN · Stochastic Bandit | greedy action | Helped |
| Add a LinUCB bonus term scaled by 0.25 (ablation) | Neural LinUCB · UCI contextual bandit benchmarks | performance | Helped |
| Scale SquareCB γ by 10 relative to the indicated proportional setting (ablation) | Neural SquareCB · UCI contextual bandit benchmarks | performance | Helped |
Surprising findings
- Even with LSTM history summarization, the agent did not reach the optimal CartPole return, which the authors attribute to needing more tuning.
Limitations
- Only model-free methods supported in this version; offline evaluation and model-based RL deferred to future versions.
- Benchmarks are preliminary sanity checks with untuned hyperparameters and few seeds (5 runs).
- Code release was planned but not yet available at time of writing.
Experimental setup
Datasets: MuJoCo continuous control tasks UCI repository datasets CartPole (partially observable variant) DeepSea Stochastic Bandit
Benchmarks: MuJoCo UCI contextual bandit datasets partially observable CartPole DeepSea Stochastic Bandit
Metrics: training return cumulative reward / return greedy action selection
Baselines: RLlib Stable-Baselines3 Tianshou CleanRL d3rlpy MushroomRL TorchRL Vowpal Wabbit
Builds on: DQN QR-DQN Bootstrapped DQN SquareCB LinUCB
Closest papers by meaning
- Preference-Guided Diffusion for Multi-Objective Offline Optimization (NeurIPS 2025 · similarity 0.85)
- Learning Configurations for Data-Driven Multi-Objective Optimization (ICML 2025 · similarity 0.85)
- PSMGD: Periodic Stochastic Multi-Gradient Descent for Fast Multi-Objective Optimization (AAAI 2025 · similarity 0.85)
- FERERO: A Flexible Framework for Preference-Guided Multi-Objective Learning (NeurIPS 2024 · similarity 0.85)
- Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning (AAAI 2024 · similarity 0.84)
- Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.