Research map / Probabilistic and scientific modeling / Molecules and proteins
Approximating mutual information of high-dimensional variables using learned representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes latent MI (LMI) approximation, which learns low-dimensional representations with a cross-predictive autoencoder and then applies the KSG estimator to estimate mutual information between high-dimensional variables.
Problem
Estimating mutual information between high-dimensional variables is infeasible for existing nonparametric and variational estimators at realistic sample sizes because sample requirements scale with dimensionality.
Approach
Train a pair of autoencoders with cross-prediction losses (each variable's encoder used to reconstruct the other variable), which upper-bounds conditional entropy, then apply the KSG nearest-neighbor estimator to held-out low-dimensional latent codes, so MI is estimated in a learned 16-dimensional space.
Key results
- LMI gives more absolutely and relatively accurate MI estimates than alternatives in 83% and 87% of tested synthetic settings, remaining stable up to 5e3 ambient dimensions where KSG, MINE, and InfoNCE fail beyond d≈13, 25, 37 respectively.
- LMI estimates MI between kinase-target and ligand-receptor protein embeddings as 0.8 and 2.8 bits, versus near-zero for shuffled data, and density-ratio classifiers achieve mean AUC-ROC 0.87 (kinase-target) and 0.74 (ligand-receptor).
- LMI estimates I(X2;X6) ≈ 0.31 bits and I(X6;X6') ≈ 0.98 bits in hematopoietic lineage-traced data, indicating non-Markovian gene expression dynamics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn low-dimensional representations with LMI | MI estimation on synthetic Gaussians with low-dimensional dependence · Synthetic Gaussians, d=10 to d=5 · 10^3, k=1 to k=9 | absolute accuracy: 83% | Helped |
| learn low-dimensional representations with LMI | MI estimation on synthetic Gaussians with low-dimensional dependence · Synthetic Gaussians, d=10 to d=5 · 10^3, k=1 to k=9 | relative accuracy: 87% | Helped |
| learn representations with LMI | MI estimation as ambient dimensionality increases while k=1 · Synthetic Gaussians, k=1, ε=0.1 bits | samples required for accurate estimation: 4 · 10^3 | Helped |
| increase the dependence dimensionality to k=d | LMI · Synthetic Gaussians, k=d | convergence behavior | Hurt |
| replace cross-predictive regularization with MINE regularization (ablation) | LMI representation learning · Synthetic Gaussian benchmarks | absolute and relative accuracy | Hurt |
| replace cross-predictive regularization with InfoNCE regularization (ablation) | LMI representation learning · Synthetic Gaussian benchmarks | absolute and relative accuracy | Hurt |
| use latent KSG instead of latent InfoNCE for estimating MI (ablation) | InfoNCE-regularized LMI representations · Synthetic Gaussian, d=100, k=1, N=5 · 10^3 | convergence to ground-truth MI: 1 bit | Helped |
| increase MINE and InfoNCE critic complexity (ablation) | Variational MI estimators · Synthetic Gaussians with 1-dimensional dependence | estimation quality | No clear effect |
| use high-complexity critics for MINE and InfoNCE (ablation) | Neural MI estimators · MNIST benchmark | estimation quality | Hurt |
| use LMI rather than MINE or InfoNCE | MI estimation on resampled protein embeddings · Resampled protein embeddings | rank correlation with ground truth: below 0 | Helped |
| use LMI rather than KSG | MI estimation on resampled real datasets · Resampled MNIST and protein embeddings | absolute accuracy | Helped |
| apply LMI density-ratio estimates to classify protein interactions | ProtTrans5 embeddings · Kinase-target and ligand-receptor interactions | AUC-ROC: 0.87 and 0.74 | Helped |
Surprising findings
- Variational bound estimators (MINE, InfoNCE) nearly always fail to estimate nonzero MI on resampled protein embeddings, with InfoNCE giving negative rank correlation.
- Increasing critic complexity did not improve variational estimator quality, suggesting larger critics are unsuitable for small sample sizes.
Limitations
- LMI fails when the dependence structure has dimensionality larger than the learned representation (16 dims), e.g. k approaching d.
- Inherits KSG limitations, failing for near-deterministic relationships between variables.
- Few theoretical properties of LMI are derived; empirical validation is the main support.
Experimental setup
Datasets: synthetic multivariate Gaussians binary MNIST subset ProtTrans5 embeddings of E. coli and A. thaliana proteins OmniPath kinase-target and ligand-receptor interactions lineage-traced hematopoietic stem cell scRNA-seq
Benchmarks: synthetic multivariate Gaussians with ground truth MI resampled MNIST MI benchmark resampled protein embedding MI benchmark OmniPath protein interaction prediction (AUC-ROC)
Metrics: mean-squared error Kendall tau rank correlation AUC-ROC runtime samples to convergence
Baselines: KSG MINE InfoNCE sliced MI
Builds on: KSG estimator MINE InfoNCE autoencoders
Closest papers by meaning
- Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning (NeurIPS 2025 · similarity 0.84)
- A Generic Family of Graphical Models: Diversity, Efficiency, and Heterogeneity (ICML 2025 · similarity 0.84)
- On the Recoverability of Causal Relations from Bulk Gene Expression Data (NeurIPS 2026 · similarity 0.84)
- Learning Molecular Representation in a Cell (ICLR 2025 · similarity 0.84)
- A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features (ICML 2025 · similarity 0.84)
- A Simple and Comprehensive Benchmark for Single-Cell Transcriptomics (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.