Research map / Learning theory and algorithms / Sample complexity
Learning Parametric Distributions from Samples and Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies when preference feedback (log-probability-based preferences over sample pairs) improves parameter estimation for continuous parametric distributions, showing M-estimators gain asymptotic variance and a constraint-based estimator achieves O(1/n) error under deterministic preferences, matched by a lower bound.
Problem
Parameter estimation for continuous parametric distributions when the learner observes sample pairs plus preferences (from log-probability rewards) depending on the same unknown parameter; whether preferences improve estimation over sample-only MLE.
Approach
Under deterministic preferences, the preference labels impose hard feasibility constraints on the parameter; estimators that minimize the empirical 0-1 preference loss (or minimize sample NLL within the feasible set) achieve an accelerated O(1/n) rate, versus the Theta(1/sqrt(n)) rate of M-estimators, with a matching minimax lower bound.
Key results
- Under deterministic preferences, the constraint-based estimators achieve estimation error O(1/n) for Laplace and Gaussian (d=1) vs Theta(1/sqrt(n)) for M-estimators.
- A minimax lower bound Omega(min{alpha(k) sqrt(k)/n, sqrt(k/n)}) matches the O(1/n) rate up to dimension and problem constants.
- Covariance gap between stochastic and deterministic M-estimators is mild, e.g. Delta values about 0.16 vs 0.08 for Laplace(0,1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add stochastic preference feedback to sample-only estimation | Stochastic preference MLE · Gaussian and Laplace distributions with log-probability preferences | asymptotic variance | Helped |
| use deterministic rather than stochastic preferences | Preference-based M-estimator · Gaussian and Laplace distributions with log-probability preferences | asymptotic variance | Helped |
| use stochastic preference MLE instead of sample-only MLE | Gaussian and Laplace distributions with log-probability preferences · Asymptotic estimation | asymptotic variance | Helped |
| use deterministic preference M-estimation instead of sample-only MLE | Gaussian distributions · Finite-sample estimation | estimation error | Helped |
| use stochastic preference MLE instead of sample-only MLE | Gaussian distributions · Finite-sample estimation | estimation error | Hurt |
| constrain the sample-only MLE to the deterministic-preference feasible set | DP MLE · Gaussian distributions | estimation error | Helped |
| use a feasible-set estimator instead of DP MLE | Arbitrary feasible estimator and worst-case feasible estimator · Gaussian distributions, larger dimensions | estimation error | Hurt |
| increase dimension at fixed sample size | Arbitrary feasible estimator and worst-case feasible estimator · Gaussian distributions | estimation error | Hurt |
| use deterministic preference constraints rather than M-estimation | Feasible-set estimators · Gaussian distributions | estimation error rate: \cO(1/\sqrt{n}) -> \cO(1/n) | Helped |
Surprising findings
- Stochastic-preference M-estimator performs worse than the deterministic-preference logistic estimator for finite samples despite lower asymptotic variance.
- Dimensionality gap between upper bound O(A k/n) and lower bound Omega(alpha sqrt(k)/n) remains open.
Limitations
- Assumptions (identifiability, linearization validity, informative preferences, positive density) are restrictive and hard to verify.
- Dimensionality gap between upper and lower bounds.
- Results do not directly apply to realistic LLM training; discrete distributions not covered.
Experimental setup
Datasets: synthetic Gaussian samples synthetic Laplace samples
Metrics: estimation error ||theta_hat - theta*|| asymptotic variance minimax lower bound
Baselines: sample-only MLE stochastic preference MLE logistic-loss preference MLE (lle)
Builds on: Bradley-Terry model M-estimation theory Assouad's lemma
Closest papers by meaning
- Statistical Rejection Sampling Improves Preference Optimization (ICLR 2024 · similarity 0.87)
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.87)
- PIPA: Preference Alignment as Prior-Informed Statistical Estimation (ICML 2025 · similarity 0.87)
- Better Estimation of the Kullback--Leibler Divergence Between Language Models (NeurIPS 2025 · similarity 0.87)
- BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling (NeurIPS 2024 · similarity 0.87)
- Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.