Research map / Reinforcement learning / Policy gradient and actor-critic

Neural-Kernel Conditional Mean Embeddings

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a neural-network-parameterized conditional mean embedding (NN-CME) that replaces Gram matrix inversion with an NN trained by a kernel-based loss, with iterative or joint bandwidth optimization, applied to conditional density estimation and distributional RL.

Problem

Kernel conditional mean embeddings face cubic cost from Gram matrix inversion, fixed features limiting expressiveness, and difficulty selecting output-kernel hyperparameters because objective values are not comparable across kernel settings.

Approach

Replace the CME weight vector beta(x) with an NN output f(x;theta) over M fixed location points eta_a, trained with an RKHS-norm loss using a Gaussian density kernel; the bandwidth is tuned either iteratively via an L2 density-error objective or jointly, using an upper bound showing the RKHS norm at sqrt(2)sigma is bounded by that at sigma. For RL, the same parameterization is trained with an MMD

Key results

• On toy datasets, Proposal-Joint achieves WAS1 of 5.67 on Bimodal and 4.68 on Skewed, versus 6.83 for NF and 6.59 for CARD on Bimodal.
• On UCI QICE, Proposal-Joint or Proposal-Iterative achieves the best or near-best result on several datasets, e.g., Naval 6.81 vs 8.34 for MDN, but CARD is best on Naval at 0.80.
• In RL, the proposed method outperforms CDQN and MMDQN on the classic control tasks, with the authors reporting clearly better performance on MountainCar and faster convergence on CartPole.

Measured effects reported in the paper

Surprising findings

• MMDQN was unstable on classic control environments despite improved performance on Atari, which the authors attribute to reward density differences.
• The median heuristic for the output kernel bandwidth in DF performs poorly, while a fixed bandwidth of 0.1 is competitive, showing bandwidth selection matters greatly.

Limitations

• Experiments are limited to one-dimensional outputs; multidimensional outputs are left for future work.
• Location parameters eta_a are fixed rather than learned, which may limit expressiveness.
• RL experiments are limited to three classic control environments without Atari evaluation and without common RL enhancements such as double Q-learning.

Experimental setup

Datasets: Bimodal toy Skewed toy Ring toy UCI Boston UCI Concrete UCI Energy UCI Kin8nm UCI Naval UCI Power UCI Protein UCI Year CartPole-v1

Benchmarks: Bimodal Skewed Ring UCI regression datasets CartPole-v1 Acrobot-v1 MountainCar-v0

Metrics: Wasserstein-1 distance (WAS1) QICE RMSE mean cumulative reward

Baselines: DF-med DF-0.1 MDN Neural Spline Flow CARD CDQN MMDQN DQN

Builds on: deep feature approach categorical DQN MMD-FUSE conditional mean embedding MMDQN

Closest papers by meaning

• Conditional Distribution Compression via the Kernel Conditional Mean Embedding (NeurIPS 2025 · similarity 0.85)
• Kernel Quantile Embeddings and Associated Probability Metrics (ICML 2025 · similarity 0.85)
• On Statistical Learning Theory for Distributional Inputs (ICML 2024 · similarity 0.84)
• Testing Conditional Mean Independence Using Generative Neural Networks (ICML 2025 · similarity 0.84)
• Learnware Specification via Label-Aware Neural Embedding (AAAI 2025 · similarity 0.83)
• Learning to Embed Distributions via Maximum Kernel Entropy (NeurIPS 2024 · similarity 0.83)