Research map / Probabilistic and scientific modeling / Variational and generative sampling
Estimating Unknown Population Sizes Using the Hypergeometric Distribution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes estimating unknown total and category sizes of under-sampled discrete populations by maximizing a (gamma-relaxed) hypergeometric likelihood, embedded in a VAE for mixtures of distributions, and validates on simulations, reading-passage complexity, and spike-in single-cell data.
Problem
Estimating discrete distributions when both total population size and category sizes are unknown, from under-sampled sampling-without-replacement count data, including mixtures of distributions.
Approach
Treat category sizes as learnable parameters of a hypergeometric likelihood, relaxed to continuous values via the gamma function, and optimize it by gradient descent; embed this in a VAE whose decoder outputs hypergeometric category sizes conditioned on a continuous latent variable, with a violation penalty enforcing estimates at least the observed counts.
Key results
- On simulated mixtures the hypergeometric VAE achieved ARI 1.00 versus 0.45 (multinomial) and 0.45 (Poisson) in a 3-distribution, 10^3-category setting, with MPE 2.8% versus 26.1% for Poisson.
- On CLEAR, the inferred latent total token count correlates strongly with readability indices, whereas the raw token count has near-zero association.
- On SPIKE, the model estimated the synthetic RNA total at roughly 30,000 per well, matching the authors' estimate, and a median human RNA total of 260,000, near a previous estimate of 200,000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the hypergeometric likelihood instead of the multinomial likelihood | VAE for count estimation and clustering · Simulated: 3 distributions, 2 unique; 10^3 categories | ARI: 0.45 ± 0.01 -> 1.00 ± 0.01 | Helped |
| Use the hypergeometric likelihood instead of the Poisson likelihood | VAE for count estimation · Simulated: 3 distributions, 2 unique; 10^3 categories | MPE: 26.1 ± 0.1% -> 2.8 ± 0.3% | Helped |
| Use the hypergeometric likelihood instead of the Poisson likelihood | VAE for count estimation · Simulated: 3 distributions, 2 unique; 10^3 categories | MAE: 3471 ± 2 -> 416 ± 49 | Helped |
| Use the hypergeometric likelihood instead of the multinomial likelihood | VAE for clustering · Simulated: 10 distributions, 9 unique; 10^3 categories | ARI: 0.89 ± 0.01 -> 0.99 ± 0.01 | Helped |
| Use the hypergeometric likelihood instead of the Poisson likelihood | VAE for count estimation · Simulated: 10 distributions, 9 unique; 10^3 categories | MPE: 6.5 ± 0.1% -> 1.6 ± 0.1% | Helped |
| Use the hypergeometric likelihood instead of the Poisson likelihood | VAE for count estimation · Simulated: 10 distributions, 9 unique; 10^3 categories | MAE: 7309 ± 43 -> 225 ± 6 | Helped |
| Use the hypergeometric likelihood instead of the multinomial likelihood | VAE for clustering · Simulated: 10 distributions; 10 categories | ARI: 1.00 ± 0.01 -> 1.00 ± 0.01 | No clear effect |
| Use the hypergeometric likelihood instead of the Poisson likelihood | VAE for count estimation · Simulated: 10 distributions; 10 categories | MPE: 6.3 ± 0.1% -> 5.2 ± 1.5% | Helped |
| Use the hypergeometric likelihood instead of the Poisson likelihood | VAE for count estimation · Simulated: 10 distributions; 10 categories | MAE: 63 ± 1 -> 52 ± 15 | Helped |
| Use the hypergeometric likelihood instead of the multinomial likelihood | VAE for clustering · Simulated: 3 distributions, 2 unique; 10^4 categories | ARI: 0.45 ± 0.01 -> 1.00 ± 0.01 | Helped |
| Use the hypergeometric likelihood instead of the Poisson likelihood | VAE for count estimation · Simulated: 3 distributions, 2 unique; 10^4 categories | MPE: 23.3 ± 0.8% -> 3.9 ± 0.9% | Helped |
| Use the hypergeometric likelihood instead of the Poisson likelihood | VAE for count estimation · Simulated: 3 distributions, 2 unique; 10^4 categories | MAE: 3744 ± 23 -> 573 ± 152 | Helped |
Surprising findings
- Raw total token count of reading passages shows near-zero association with readability, while the model's inferred latent vocabulary size correlates strongly.
Limitations
- The hypergeometric likelihood is relaxed with the gamma function, so latent counts are continuous and rounding is needed for discrete answers.
- Estimation for the SPIKE experiment relies on a very small number of observations (1126 cells) and no true human transcript ground truth.
- Some simulated settings show high variance or failures (e.g., ARI 0.51 and MAE 8767 in one 10-distribution setting).
Experimental setup
Datasets: simulated count data CLEAR Corpus SPIKE single-cell spike-in dataset
Benchmarks: simulated hypergeometric count datasets CLEAR readability indices SPIKE spike-in RNA ground truth
Metrics: mean absolute error median percentage error adjusted Rand index Manhattan error Pearson correlation
Baselines: multinomial likelihood VAE Poisson likelihood VAE
Builds on: variational autoencoder multinomial VAE for collaborative filtering capture-recapture estimation
Closest papers by meaning
- Approximating mutual information of high-dimensional variables using learned representations (NeurIPS 2024 · similarity 0.83)
- Efficient Mixture Learning in Black-Box Variational Inference (ICML 2024 · similarity 0.82)
- Generative Modeling under Non-Monotone MAR Missingness via Approximate Wasserstein Gradient Flows (NeurIPS 2026 · similarity 0.82)
- Robust group and simultaneous inferences for high-dimensional single index model (NeurIPS 2024 · similarity 0.82)
- Learning With Multi-Group Guarantees For Clusterable Subpopulations (ICML 2025 · similarity 0.82)
- A Unified Framework for Variable Selection in Model-Based Clustering with Missing Not at Random (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.