Research map / Large language models / Agents, code and math
Dynamic Evaluation of Large Language Models by Meta Probing Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Meta Probing Agents (MPA), an LLM-agent dynamic evaluation protocol that rewrites benchmark questions via psychometric-inspired principles (language understanding, problem solving, domain knowledge) with a judge agent checking consistency, revealing performance drops suggestive of contamination.
Problem
Static public benchmarks are vulnerable to data contamination and provide only overall scores, while existing dynamic evaluation methods are task-specific and cannot support fine-grained multifaceted ability analysis.
Approach
Use GPT-4-Turbo as probing agents that transform benchmark questions according to five psychometric principles grouped under three basic abilities, with a judge agent that validates each transformed question with a Yes/No consistency check, enabling dynamic sample generation and modular ability analysis.
Key results
- GPT-4-Turbo accuracy on MMLU dropped from 84.40 to 68.86 on MPA probing benchmark (-15.54 points), and all evaluated models degraded on the probing benchmarks.
- Human verification of generated questions reported 94% equivalence and 97% correctness on average.
- Fine-tuning GPT-3.5-Turbo on MPA-generated data gave about 2% average improvement on MMLU and ARC-C according to the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate on MPA-generated samples instead of the vanilla benchmark | GPT-4-Turbo · MMLU | accuracy: 84.40 -> 68.86 | Hurt |
| evaluate on MPA-generated samples instead of the vanilla benchmark | GPT-4-Turbo · GSM8K | accuracy: 95.22 -> 88.50 | Hurt |
| evaluate on MPA-generated samples instead of the vanilla benchmark | GPT-4-Turbo · ARC-C | accuracy: 96.16 -> 84.67 | Hurt |
| evaluate on MPA-generated samples instead of the vanilla benchmark | GPT-4-Turbo · BBH (partial) | accuracy: 88.53 -> 87.78 | Hurt |
| apply principle 1, paraphrase questions, alone (ablation) | MPA with GPT-4 · MMLU | accuracy: 84.40 -> 78.43 | Hurt |
| apply principle 2, paraphrase choices, alone (ablation) | MPA with GPT-4 · MMLU | accuracy: 84.40 -> 81.48 | Hurt |
| apply principle 3, permute choices, alone (ablation) | MPA with GPT-4 · MMLU | accuracy: 84.40 -> 80.27 | Hurt |
| apply principle 4, add extra context, alone (ablation) | MPA with GPT-4 · MMLU | accuracy: 84.40 -> 81.42 | Hurt |
| apply principle 5, add a new choice, alone (ablation) | MPA with GPT-4 · MMLU | accuracy: 84.40 -> 83.73 | Hurt |
| apply principle 1, paraphrase questions, alone (ablation) | MPA with GPT-4 · ARC-C | accuracy: 96.16 -> 94.11 | Hurt |
| apply principle 2, paraphrase choices, alone (ablation) | MPA with GPT-4 · ARC-C | accuracy: 96.16 -> 94.28 | Hurt |
| apply principle 3, permute choices, alone (ablation) | MPA with GPT-4 · ARC-C | accuracy: 96.16 -> 93.69 | Hurt |
Surprising findings
- Chain-of-thought and in-context learning did not effectively boost performance on the probing benchmarks.
- Larger models showed stronger correlations between basic abilities, described as an implicit Matthew effect with model size.
Limitations
- Only four datasets (plus three BBH subtasks) were used, limiting breadth of tasks.
- Judge agent sometimes misses meaning drift; weaker probing or judging models (GPT-3.5, Gemini-Pro) were insufficient, so GPT-4-class models are required.
- Evaluation relies on GPT-4-Turbo as agent, which may introduce bias and cost.
Experimental setup
Datasets: MMLU ARC-Challenge GSM8K BIG-Bench Hard (formal fallacies, object counting, temporal sequences)
Benchmarks: MMLU GSM8K ARC-Challenge BBH (partial)
Metrics: accuracy standard deviation relative effectiveness Pearson correlation Spearman correlation Kendall correlation
Baselines: GPT-4-Turbo GPT-3.5-Turbo Gemini-Pro Yi-34b-chat Mixtral-8x7b-Instruct Llama2-70b-chat DyVal NPHardEval
Builds on: DyVal NPHardEval HELM
Closest papers by meaning
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.89)
- MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.89)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks (ICLR 2024 · similarity 0.89)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.88)
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.