Research map / Large language models / Language, safety and interpretability
Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GETA, a generative evolving testing framework that jointly trains a variational IRT model and an LLaMA-3-8B item generator to dynamically create difficulty-tailored value-alignment test items and evaluate LLMs on bias, ethics, and toxicity.
Problem
Static benchmarks for measuring LLM value alignment suffer from evaluation chronoeffect: data leakage into training data and saturation as models rapidly improve, leading to overestimated value conformity.
Approach
Integrates computerized adaptive testing (IRT-based item selection) with automatic item generation by using an LLM item generator conditioned on IRT item parameters, trained jointly with a variational IRT estimator under an ELBO, and iteratively evolved by collecting items beyond the difficulty boundary to fine-tune the generator.
Key results
- GETA achieves overall validity 0.8753 in the ablation, versus 0.7587 for removing both VIRT and the item generator (w/o Both, i.e., CAT-like).
- GETA reaches Va-L 0.8897 compared with 0.6433 for the variant without VIRT-AIG components, and removing VIRT drops Va-L to 0.4309.
- GETA evaluates with roughly 150 adaptive test items and finds LLaMA-2-70B more biased than LLaMA-2-7B (80.91% vs 39.67% biased responses), consistent with the Enkrypt leaderboard.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use variational inference instead of MLE for VIRT (ablation) | GETA · Concurrent validity | Overall: 0.4876 -> 0.8753 | Helped |
| use a generated item pool instead of a static item pool (ablation) | GETA · Concurrent validity | Overall: 0.8585 -> 0.8753 | Helped |
| use VIRT and an item generator instead of removing both (ablation) | GETA · Concurrent validity | Overall: 0.7587 -> 0.8753 | Helped |
| iteratively update the item generator during testing (ablation) | GETA · Concurrent validity | Va-L: 0.8664 -> 0.8897 | Helped |
| use a Transformer instead of an RNN for VIRT (ablation) | GETA · Concurrent validity | Overall: 0.7784 -> 0.8753 | Helped |
Surprising findings
- NCAT ranks GPT-4 last in ethics and toxicity, showing inconsistent conclusions across value types.
- Static evaluation and CAT rank Orca-2-13B above LLaMA-2-7B-Chat on bias, which is counterintuitive.
Limitations
- Only the 2PL IRT model is used, which may limit the measurement quality for value conformity.
- Evaluated on a small set of examinee LLMs (eight), and the sensitivity to examinee group size is limited.
- Dual-use risk: the item generator could be used to produce harmful test items at scale.
Experimental setup
Datasets: BBQ CrowS-Pairs RedditBias ETHICS Anthropic red team BAD Do-Not-Answer HarmfulQ HarmfulQA RealToxicityPrompts
Benchmarks: Enkrypt AI LLM Safety Leaderboard DecodingTrust Leaderboard held-out i.i.d GETA items OOD AdvPromptSet items
Metrics: Value Conformity Concurrent validity (Pearson correlation) Empirical probability Pearson correlation
Baselines: Static Evaluation CAT NCAT GPTFuzzer SAP
Builds on: Computerized Adaptive Testing Item Response Theory (2PL) Automatic Item Generation NCAT GPTFuzzer
Closest papers by meaning
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ICML 2026 · similarity 0.90)
- ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation (NeurIPS 2024 · similarity 0.90)
- DENEVIL: TOWARDS DECIPHERING AND NAVIGATING THE ETHICAL VALUES OF LARGE LANGUAGE MODELS VIA INSTRUCTION LEARNING (ICLR 2024 · similarity 0.90)
- CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses (NeurIPS 2024 · similarity 0.90)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.89)
- Reliable and Efficient Amortized Model-based Evaluation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.