Research map / Reinforcement learning / Offline RL

Meta-Black-Box-Optimization through Offline Q-function Learning

ICML 2025 · Reinforcement learning › Offline RL · approach: Diffusion models for offline decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Q-Mamba, an offline RL framework for meta-black-box optimization that decomposes the Q-function over hyper-parameters and uses a Mamba-based Q-learner trained with a conservative decomposed Q-loss on a mixed exploration/exploitation dataset.

Problem

Online RL-based MetaBBO for dynamic algorithm configuration is sample-inefficient, and learning policies over the massive joint hyper-parameter space of BBO algorithms is difficult.

Approach

Turn DAC into a long sequence where each hyper-parameter is one decision step with a per-dimension Q-function, train it offline with a compositional Q-loss (Bellman TD plus CQL-style regularization on unselected bins), and use a Mamba backbone for efficient long-sequence learning.

Key results

• Q-Mamba achieves average performance 9.889E-01 on Alg0, 9.973E-01 on Alg1 and 9.950E-01 on Alg2, matching or exceeding online baselines RLPSO, LDE and GLEET.
• Q-Mamba trains in about 13h versus 25-28h for online baselines, under half the training budget.
• Ablation shows removing the conservative term (lambda=0) lowers performance to 9.833E-01 with beta=10, and mixing exploration and exploitation data at mu=0.5 gives the best result.

Measured effects reported in the paper

Surprising findings

• Offline Q-Mamba matches or beats online MetaBBO baselines despite learning from a fixed dataset.
• Increasing the number of action bins beyond 16 does not consistently improve performance.

Limitations

• The learned policy must be retrained for each low-level BBO algorithm.
• Evaluation is limited to a small set of algorithms and BBOB functions, with only 8 test instances.
• Claims of superiority over Transformer-based Q-learning are attributed to Mamba architecture without isolating the effect.

Experimental setup

Datasets: CoCo BBOB testsuite E&E offline trajectory dataset (constructed)

Benchmarks: BBOB test instances (8 held-out functions) neuroevolution MuJoCo continuous control tasks

Metrics: accumulated performance improvement training time inference time

Baselines: RLPSO LDE GLEET Decision Transformer DeMa QDT QT Q-Transformer

Builds on: CQL Q-Transformer Mamba DQN DE

Closest papers by meaning

• MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimization (NeurIPS 2025 · similarity 0.89)
• $\texttt{MetaDistill}$: Unlocking the Performance Ceiling for Pretrained Optimizers (ICML 2026 · similarity 0.89)
• ROOT: Rethinking Offline Optimization as Distributional Translation via Probabilistic Bridge (NeurIPS 2025 · similarity 0.86)
• SYMBOL: Generating Flexible Black-Box Optimizers through Symbolic Equation Learning (ICLR 2024 · similarity 0.86)
• ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning (AAAI 2025 · similarity 0.85)
• Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings (ICML 2025 · similarity 0.85)