Research map / Probabilistic and scientific modeling / Variational and generative sampling
Learning to Embed Distributions via Maximum Kernel Entropy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes unsupervised learning of a data-dependent distribution kernel by maximizing the quantum Rényi-2 entropy of the covariance operator embedding of datasets of distributions (MDKE), with an encoder mapping samples to the hypersphere.
Problem
Choosing a suitable kernel between distributions for distribution regression is nontrivial, and learning data-dependent distribution kernels without labels has received little attention.
Approach
Learn an encoder f_theta mapping samples to the hypersphere by maximizing the second-order Rényi quantum entropy of the covariance operator embedding of a dataset of distributions, where the Gram matrix of a Gaussian distribution kernel is used to compute the entropy; theory links this to distributional variance and uniform configurations on the sphere.
Key results
- On flow cytometry Tissue classification, MDKE with one configuration reached 98.89% accuracy vs 93.07% for GMM-FV, the best baseline.
- On flow cytometry Leukemia classification, the best baseline SW2 reached 95.30% and MDKE reached 94.57%, so MDKE did not outperform there.
- On MNIST, unsupervised pre-training improved the random-initialization accuracy of 85.0% to a plateau of 92.15%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain the latent embeddings with MDKE instead of random initialization (ablation) | MNIST distribution classification · MNIST | accuracy: 85.0% -> 92.15% | Helped |
| optimize distributional variance directly instead of MDKE (ablation) | Flow Cytometry classification · Flow Cytometry Tissue | accuracy: 98.89% -> 79.47% | Hurt |
| optimize distributional variance directly instead of MDKE (ablation) | Flow Cytometry classification · Flow Cytometry Leukemia | accuracy: 94.57% -> 91.82% | Hurt |
| add determinant-based regularization to MDKE (ablation) | MDKE sentence representation learning · 20 Newsgroups | eigenvalue stability | Helped |
Surprising findings
- The paper reports that directly optimizing distributional variance with SGD gave significantly weaker results than optimizing the entropy objective.
- On the Leukemia task, several MDKE configurations and the Distributional Variance ablation underperformed the best kernel baselines, with the MDKE configuration without regularization at 89.50%.
Limitations
- Computational cost scales with the number of distributions (O(N^2) Gram matrix) and needs many samples per distribution, so best suited to few distributions with many samples.
- Optimization is non-convex in the encoder parameters and small eigenvalues of the kernel matrix can collapse, requiring an ad hoc regularizer.
- Image and text experiments treat inputs as histograms or word sets, which the authors acknowledge are less powerful than domain-specific models.
Experimental setup
Datasets: Flow cytometry dataset (Tissue and Leukemia labels) MNIST Fashion-MNIST 20 Newsgroups
Benchmarks: Flow cytometry Tissue classification Flow cytometry Leukemia classification MNIST Fashion-MNIST 20 Newsgroups (5 classes)
Metrics: classification accuracy variance over repeated runs
Baselines: GMM-FV Sliced Wasserstein-1 kernel Sliced Wasserstein-2 kernel MMD kernels distributional variance
Builds on: kernel mean embedding Rényi quantum entropy distribution regression sliced Wasserstein kernel Fisher kernel
Closest papers by meaning
- On Statistical Learning Theory for Distributional Inputs (ICML 2024 · similarity 0.89)
- Kernel Quantile Embeddings and Associated Probability Metrics (ICML 2025 · similarity 0.88)
- Supervised Distributional Reduction via Optimal Transport and Dependence Maximization (NeurIPS 2026 · similarity 0.86)
- On the Consistency of Kernel Methods with Dependent Observations (ICML 2024 · similarity 0.86)
- Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors (ICLR 2025 · similarity 0.85)
- Conditional Distribution Compression via the Kernel Conditional Mean Embedding (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.