Research map / Large language models / Language, safety and interpretability
Evaluating Large Language Models at Evaluating Instruction Following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LLMBar, a manually curated meta-evaluation benchmark of 419 instruction-following output pairs, shows LLM evaluators struggle on its adversarial subsets, and proposes prompting strategies (Rules, Metrics, Swap) that narrow the gap to humans.
Problem
Existing meta-evaluation benchmarks for LLM evaluators use subjective, noisy human preference labels, so they cannot reliably show whether evaluators detect objective instruction-following quality versus superficial appeal.
Approach
Build a meta-evaluation set where every instance has an objectively better output for instruction following, curated by authors, with adversarial subsets whose dispreferred outputs have superficial appeal; also propose prompting strategies (Rules, Metrics, Reference, Swap+CoT) to improve evaluators.
Key results
- Best GPT-4 evaluator (Metrics+Reference*) reaches 82.8% average accuracy on LLMBar Adversarial versus 95% expert human agreement.
- ChatGPT, LLaMA-2-70B-Chat, and Falcon-180B evaluators reach near or below chance on the Adversarial set under vanilla prompting, while the proposed strategies add about 10 points for GPT-4.
- Expert human annotators agree 94% on sampled LLMBar instances, compared with 63% for MT-Bench and 71.7% for FairEval.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Rules to Vanilla (ablation) | GPT-4 evaluator · LLMBar Adversarial average | accuracy: 73.1 -> 80.8 | Helped |
| Add self-generated metrics and a self-generated reference, together with Rules (ablation) | GPT-4 evaluator · LLMBar Adversarial average | accuracy: 73.1 -> 82.8 | Helped |
| Add self-generated metrics and a self-generated reference, together with Rules (ablation) | GPT-4 evaluator · LLMBar Natural | accuracy: 93.5 -> 96.0 | Helped |
| Add Chain-of-Thought reasoning to Rules (ablation) | GPT-4 evaluator · LLMBar Adversarial average | accuracy: 80.8 -> 76.6 | Hurt |
| Add Swap and Synthesize to Rules (ablation) | GPT-4 evaluator · LLMBar Adversarial average | positional agreement rate: 91.0 -> 96.0 | Helped |
Surprising findings
- Chain-of-thought reasoning falls short of improving evaluators on the Adversarial set, as its reasoning often favors superficially appealing outputs.
- Existing meta-evaluation benchmarks show similar performance across LLMs and prompts, while LLMBar reveals a clear gap between weak and strong evaluators.
Limitations
- Adversarial subset GPTOut was generated by GPT-4, which may give GPT-4-based evaluators an unfair advantage.
- Benchmark is manually curated and limited in diversity; only single-round interactions are covered.
- Focuses only on instruction following, not factuality or toxicity.
Experimental setup
Datasets: AlpacaFarm LLMEval^2 Alpaca OpenAssistant ShareGPT FairEval MT-Bench
Benchmarks: LLMBar FairEval LLMEval^2 MT-Bench
Metrics: accuracy positional agreement rate
Baselines: Vanilla CoT ChatEval AlpacaFarm reward-model-sim AlpacaFarm reward-model-human SteamSHP-flan-t5-xl
Builds on: chain-of-thought ChatEval AlpacaFarm MT-Bench
Closest papers by meaning
- DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation (AAAI 2024 · similarity 0.92)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.91)
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.90)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.90)
- XIFBench: Evaluating Large Language Models on Multilingual Instruction Following (NeurIPS 2025 · similarity 0.90)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.