Research map / Large language models / Language, safety and interpretability
Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Collects 18,972 human survey examples of how people generalize LLM correctness across question pairs from 79 MMLU/BBH tasks, models those belief changes with NLP (BERT best), and uses the model to show that larger LLMs can be less aligned with human deployment decisions under high mistake costs.
Problem
LLMs are deployed where people believe they will perform well, so evaluating them requires modeling how humans generalize from observed model responses to beliefs about other questions; existing fixed-distribution benchmarks miss this human deployment effect.
Approach
Formalizes a human generalization function that maps observed LLM correctness on one question to beliefs about correctness on another, collects a large survey dataset of such updates, predicts belief changes with NLP models, and evaluates LLM alignment via a weighted generalized accuracy that encodes deployment risk aversion.
Key results
- BERT predicts human belief change with AUC 0.81 versus 0.60 for the previous-correct baseline; BERT has the lowest NLL (0.593).
- Under a 50% deployment threshold GPT-4 has the highest weighted generalized accuracy (0.569), but at a 99% threshold it has the lowest among the evaluated models (0.379 vs 0.392 for Alpaca 7B in the table).
- GPT-3.5 turbo and GPT-4 perform worse than fine-tuned BERT and Llama-2 at predicting belief change (GPT-4 NLL 0.789 vs BERT 0.593).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BERT text features instead of the previous-correct-only baseline | human belief-change prediction · Human belief-change test set | AUC (overall): 0.599 -> 0.812 | Helped |
| Add whether the two questions are from the same task | Previous correct baseline · Human belief-change test set | AUC (overall): 0.599 -> 0.688 | Helped |
| Use fine-tuned BERT rather than fine-tuned Llama-2 (13B) | human belief-change prediction · Human belief-change test set | AUC (overall): 0.787 -> 0.812 | Helped |
| Increase Llama-2 size from 7B to 13B | human belief-change prediction · Human belief-change test set | AUC (overall): 0.755 -> 0.787 | Helped |
| Use larger LLMs for prediction rather than the smaller BERT model | human belief-change prediction · Human belief-change test set | AUC (overall): 0.812 vs. 0.787 | Hurt |
| Use GPT-4 rather than GPT-3.5 turbo for belief-change prediction | few-shot belief-change prediction · Human belief-change test set | AUC (overall): 0.537 -> 0.640 | Helped |
| Use larger LLMs for human-deployment alignment | LLM deployment evaluated against human generalizations · MMLU and BBH (50% deployment threshold) | weighted generalized accuracy | Helped |
| Use larger LLMs for human-deployment alignment | LLM deployment evaluated against human generalizations · MMLU and BBH (99% deployment threshold) | weighted generalized accuracy: 0.392 vs. 0.379 | Hurt |
| Raise deployment threshold from 50% to 99% confidence | GPT-4 alignment with human generalizations · MMLU and BBH | weighted generalized accuracy: 0.569 -> 0.379 | Hurt |
Surprising findings
- Larger, more capable LLMs (e.g., GPT-4) become less aligned with human generalizations as the cost of mistakes rises, with smaller models outperforming them in high-stakes settings.
- A small fine-tuned BERT predicts human generalization better than larger modern LLMs like GPT-4 prompted with 10 examples.
Limitations
- Human generalization function is studied only in aggregate, ignoring heterogeneity across people.
- Only single-question generalizations are collected; multi-question generalization is untested.
- Hypothetical LLMs are used in surveys, and the human generalization function may change over time.
Experimental setup
Datasets: MMLU BIG-Bench Hard
Benchmarks: MMLU BIG-Bench Hard
Metrics: negative log-likelihood AUC weighted generalized accuracy weighted binary cross-entropy calibration
Baselines: Previous correct logistic regression Same task + previous correct Fixed embeddings + XGBoost Llama-2 (7B) fine-tuned Llama-2 (13B) fine-tuned GPT-3.5 turbo 10-shot GPT-4 10-shot
Builds on: epsilon-greedy bandit sampling BERT XGBoost
Closest papers by meaning
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.90)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.90)
- Delving into the Reversal Curse: How Far Can Large Language Models Generalize? (NeurIPS 2024 · similarity 0.89)
- On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.89)
- Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.89)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.