Research map / Large language models / Language, safety and interpretability
Evaluating language models as risk scores
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces folktexts, a package that turns US Census (ACS PUMS) tabular records into natural-language prompts so LLMs act as zero-shot risk scores, and evaluates 17 LLMs for calibration and discrimination on unrealizable prediction tasks.
Problem
Standard LLM QA benchmarks are realizable and accuracy-based, so they cannot assess whether LLMs quantify outcome uncertainty when used as risk scores for underspecified prediction tasks.
Approach
Map tabular census prediction tasks to natural-language prompts and extract calibrated-evaluable risk scores from next-token probabilities of multiple-choice answers (or verbalized numeric probabilities), evaluating calibration against known population distributions rather than unique labels.
Key results
- Multiple-choice zero-shot LLM risk scores achieve high AUC (up to about 0.86 for Llama 3 70B it on ACSIncome) but high ECE for nearly all models, while LR and XGBoost reach ECE of 0.03 and 0.00.
- Base models underestimate outcome uncertainty less than instruction-tuned models; instruction-tuned models are over-confident (e.g., Llama 3 70B base average under-confidence bias 0.07 vs instruct over-confidence 0.22).
- Verbalized numeric prompting improves ECE for most instruction-tuned model/task pairs (e.g., Llama 3 70B it ACSIncome ECE 0.27 to 0.25 with AUC drop; ACSMobility ECE 0.17 to 0.05 for Llama 3 70B it) but generally reduces AUC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use numeric risk prompting instead of multiple-choice prompting | GPT 4o mini (it) · ACSIncome | ECE: 0.24 -> 0.05 | Helped |
| Use numeric risk prompting instead of multiple-choice prompting | GPT 4o mini (it) · ACSIncome | AUC: 0.85 -> 0.83 | Hurt |
| Use numeric risk prompting instead of multiple-choice prompting | instruction-tuned models · ACSIncome | ECE: 0.16 -> 0.10 | Helped |
| Use numeric risk prompting instead of multiple-choice prompting | instruction-tuned models · ACSIncome | AUC: 0.86 -> 0.84 | Hurt |
| Apply instruction-tuning | Llama 3 70B · ACSIncome | ECE: 0.20 -> 0.27 | Hurt |
| Apply instruction-tuning | Llama 3 70B · ACSIncome | AUC: 0.86 -> 0.86 | No clear effect |
| Use multiple-choice prompting rather than numeric risk prompting | LLMs, across five benchmark tasks · ACSIncome and other folktexts tasks | AUC | Helped |
| Add more input features to the prediction task | Mixtral 8x7B · ACSIncome | AUC | Helped |
Surprising findings
- Instruction-tuning polarizes multiple-choice answer distributions regardless of true data uncertainty, worsening calibration while improving AUC and accuracy.
- Gemma instruction-tuned models degenerate into predicting only positive outcomes with high confidence, and some Gemma models have worse-than-random accuracy despite above-random AUC.
Limitations
- Calibration is not sufficient for fair or safe use in consequential risk assessment; subgroup miscalibration is only surfaced, not analyzed in depth.
- Results vary substantially across tasks; LLMs underperform linear baselines on ACSMobility, so some labeled data may still be needed.
- Numeric prompting produces many tied scores, which reduces AUC.
Experimental setup
Datasets: ACS PUMS 2018 US Census microdata
Benchmarks: ACSIncome ACSEmployment ACSMobility ACSTravelTime ACSPublicCoverage
Metrics: ECE Brier score AUC accuracy confidence bias signed calibration error
Baselines: logistic regression XGBoost
Builds on: folktables
Closest papers by meaning
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.91)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.89)
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.89)
- Reliable and Efficient Amortized Model-based Evaluation (ICML 2025 · similarity 0.89)
- Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction (ICML 2025 · similarity 0.89)
- SteerConf: Steering LLMs for Confidence Elicitation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.