Research map / Large language models / Language, safety and interpretability
Reliable and Efficient Amortized Model-based Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes amortized, model-based (IRT/Rasch) LLM evaluation: a content-based difficulty predictor replaces per-question calibration, and an SFT+PPO-trained conditional question generator supports adaptive testing.
Problem
Holistic LM evaluation over large benchmarks is costly, and subset average scores are confounded by question difficulty, making cheap evaluations unreliable and question-bank calibration expensive.
Approach
Combine Rasch/IRT ability estimation with amortized calibration (regressing question difficulty from question embeddings plus dataset description) and a difficulty-conditioned question generator trained with SFT then PPO, enabling test-invariant ability estimates and Fisher-information adaptive testing over a large generated question bank.
Key results
- Adaptive (Fisher information) testing reduces queries needed for 95% reliability by 53% on average and up to 86% versus random testing across datasets.
- Amortized calibration from Llama-3-8B embeddings closely matches traditional calibration on AUC and ability-estimation metrics across all datasets.
- Rasch-score generalization AUC of 0.78 ± 0.07 versus 0.5 ± 0.07 for subset average score; SFT+PPO generator reduces difficulty-prediction error ~10x over SFT-only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a separate ability parameter for each model on each dataset rather than one ability across datasets | Rasch response model · All datasets | AUC: 0.85 -> 0.89 (train); 0.83 -> 0.87 (test) | Helped |
| Replace subset average score with Rasch ability scoring | Ability estimation on random held-out question subsets · New questions, held-out subsets | AUC: 0.5 ± 0.07 -> 0.78 ± 0.07 | Helped |
| Use adaptive Fisher-information question selection instead of random question selection | AIRBench evaluation · AIRBench | Questions needed for 95% reliability: average 53% reduction; up to 86% reduction | Helped |
| Use a large rather than small calibrated question bank for Fisher adaptive testing | Adaptive testing · AIRBench | Reliability at the same query budget: 31 queries | Helped |
| Add PPO difficulty-alignment training after SFT (ablation) | Conditional question generator · Question difficulty prediction | Prediction error: nearly 10x | Helped |
| Use Mistral 7B v0.3 embeddings instead of Llama-3-8B embeddings (ablation) | Amortized difficulty calibration · All datasets | Calibration metrics | No clear effect |
| Use Mistral 7B v0.3 as the generator base model instead of Llama-3.1-Instruct-8B (ablation) | Conditional question generator · Question difficulty generation | Difficulty prediction results | No clear effect |
| Add a discrimination parameter (2PL) or guessing parameter (3PL) to the Rasch model (ablation) | IRT response model · All datasets | AUC | No clear effect |
Surprising findings
- Question difficulty, not test-taker ability, dominates predictive power: a difficulty-only model achieves much of the AUC, since questions outnumber test takers by orders of magnitude.
- The 2PL and 3PL IRT models do not outperform the 1PL Rasch model, attributed to limited number of test takers and overfitting.
Limitations
- Generated question quality depends on training data and difficulty-predictor accuracy; generated questions may be misaligned or biased, requiring human review.
- Model responses depend on factors beyond ability and difficulty (sampling temperature, few-shot examples, chain-of-thought), which the Rasch model does not capture.
- Only dichotomously gradeable benchmarks are used; multi-valued responses are not handled.
Experimental setup
Datasets: AIR-Bench 2024 MMLU MATH GSM8K BoolQ WikiFact CivilComments TruthfulQA bAbI HELM Classic/Lite/MMLU/Thai Exam subsets
Benchmarks: AIR-Bench 2024 22 HELM-derived NLP datasets
Metrics: AUC correlation with limiting average score empirical reliability sample complexity (queries to 95% reliability) Spearman rank correlation
Baselines: average score on random subset naive response model flash-helm SFT-only question generator
Builds on: Rasch model proximal policy optimization Flash-HELM scaling laws
Closest papers by meaning
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ICML 2026 · similarity 0.91)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.90)
- TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning (AAAI 2025 · similarity 0.89)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.89)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.89)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.