Research map / Large language models / Language, safety and interpretability
Uncertainty Quantification for LLM-Based Survey Simulations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a data-driven method that adaptively selects the number of LLM-simulated survey responses to build valid confidence sets for human population parameters, tested on OpinionQA and EEDI with eight LLMs.
Problem
LLM-simulated survey responses deviate from the real human population, so naive confidence intervals from simulated samples either undercover (too many samples) or are uninformative (too few), and the right simulation sample size is unknown.
Approach
Select the simulation sample size k by controlling an empirical miscoverage proxy computed on similar real survey questions, where real-data confidence sets stand in for the unknown true parameters; the chosen k gives average-case coverage guarantees for any LLM and any confidence-set procedure, and k itself quantifies LLM-human misalignment.
Key results
- On both OpinionQA and EEDI, hypothesis tests of the miscoverage proxy exceeding alpha yield reasonably large p-values across all LLMs and alpha values, consistent with the coverage guarantee.
- Selected simulation sample sizes b_k are below 100 for all LLMs; the smallest can be in single digits, showing that generating too many synthetic samples is risky.
- On OpinionQA all LLMs clearly outperform the random baseline on b_k, while on EEDI only DeepSeek-V3 and GPT-4o outperform random; GPT-4o is best on OpinionQA and DeepSeek-V3 best on EEDI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o instead of other evaluated LLMs | LLM survey simulation · OpinionQA | selected simulation sample size | Helped |
| Use DeepSeek-V3 instead of other evaluated LLMs | LLM survey simulation · EEDI | selected simulation sample size | Helped |
| Use GPT-4o or DeepSeek-V3 instead of the random response generator | LLM survey simulation · EEDI | selected simulation sample size | Helped |
| Use an evaluated LLM instead of the random response generator | LLM survey simulation · OpinionQA | selected simulation sample size | Helped |
| Simulate subjective social opinions rather than student mathematics answers | Evaluated LLMs · OpinionQA vs EEDI | selected simulation sample size | Helped |
| Select the simulation sample size adaptively using real and simulated responses | Proposed uncertainty quantification method · OpinionQA and EEDI | miscoverage probability proxy | Helped |
Surprising findings
- On the EEDI math dataset only DeepSeek-V3 and GPT-4o beat a uniform random responder, and LLMs achieve uniformly higher b_k on opinion questions than on student-answer questions.
Limitations
- Coverage guarantee holds only on average over randomness of questions and responses, not for every instance.
- Does not explicitly minimize confidence set size; intervals may be larger than necessary.
- Relies on a CLT approximation P(theta_j <= ybar_j | psi_j) = 1/2 in the theory.
Experimental setup
Datasets: OpinionQA EEDI
Benchmarks: OpinionQA EEDI
Metrics: miscoverage probability proxy selected simulation sample size b_k p-value of coverage hypothesis test confidence interval half-width
Baselines: random response generator
Builds on: prediction-powered inference conformal inference Hoeffding's inequality
Closest papers by meaning
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.90)
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.89)
- Benchmarking LLMs via Uncertainty Quantification (NeurIPS 2024 · similarity 0.87)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.87)
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.87)
- Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.