Research map / Learning theory and algorithms / Prediction and decision losses

Sampled Estimators For Softmax Must Be Biased

NeurIPS 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Losses for decision-making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Argues that sampled estimators of the softmax normalizer must be biased, and analyzes the resulting trade-offs for large-class probabilistic models.

Problem

Computing the softmax normalization sum over all classes is prohibitively expensive when the number of classes is very large, as in NLP, contrastive learning, and recommendation systems.

Approach

Claims that any sampled estimator of the softmax must be biased, so approximations to the normalizer must accept bias in exchange for tractable cost; the paper builds its analysis and proposed estimator around this claim.

Surprising findings

• Sampled estimators of the softmax must be biased, contrary to the common expectation of unbiased sampled softmax estimation.

Experimental setup

Builds on: sampled softmax

Closest papers by meaning

• Adaptive Sampling for Efficient Softmax Approximation (NeurIPS 2024 · similarity 0.87)
• Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shifts (ICML 2024 · similarity 0.85)
• Binary Hypothesis Testing for Softmax Models and Leverage Score Models (ICML 2025 · similarity 0.85)
• Rethinking Approximate Gaussian Inference in Classification (NeurIPS 2025 · similarity 0.85)
• $\epsilon$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise (NeurIPS 2024 · similarity 0.84)
• Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learning (AAAI 2025 · similarity 0.84)