Research map / Large language models / Language, safety and interpretability
Rethinking Generative Large Language Model Evaluation for Semantic Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-examines multiple-choice QA evaluation of LLMs across 11 benchmarks and proposes RWQ-Elo, a GPT-4-judged pairwise Elo rating system over 20,772 real-world user questions.
Problem
MCQA-based accuracy is a popular but unreliable way to evaluate generative LLMs: evaluation strategies vary across models, results are sensitive to choice order, and MCQA predictions often disagree with open-ended answers, so it poorly reflects real-world usage for semantic comprehension.
Approach
Shows MCQA inconsistencies (strategy sensitivity, circular-order instability, mismatch with open-ended answers) and replaces it with a round-robin two-player Elo tournament where LLMs answer real user questions and GPT-4 judges pairwise winners, with a fast-registration scheme for adding new models.
Key results
- Open-ended MCQA with LLaMA-2-Chat-13B on filtered MMLU drops from 54.0% (Choices) to 39.7%.
- GPT-4 judge agrees with human evaluator on pairwise decisions 95% of the time on 300 RWQ questions.
- Fast-registration with N2=10 new models yields averaged rating difference of 3.6 and Kendall's tau of 0.99 versus from-scratch Elo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate LLaMA-2-Chat-13B with free-form generation and GPT-4 semantic matching instead of the Choices strategy | LLaMA-2-Chat-13B · Filtered MMLU 0-shot | accuracy: 54.0 -> 39.7 | Hurt |
| Require correct predictions across circularly reordered answer choices | LLMs evaluated with Choices or Vocab · 11 0-shot datasets | accuracy | Hurt |
| Use fast-registration rather than rerunning Elo from scratch | RWQ-Elo · RWQ fast-registration, N2=1, 3, 5, and 10 | Kendall's tau: 0.97; 0.98; 0.98; 0.99 | No clear effect |
Surprising findings
- Circular evaluation (rotating choice order) causes a notable decline for all LLMs, showing inconsistency in MCQA predictions.
- Some LLMs generate their own set of choices before answering, suggesting possible benchmark data leakage or insufficient instruction tuning.
Limitations
- Cannot evaluate all prior LLMs due to limited resources and rapid model development.
- GPT-4 judge alignment with humans measured on only 300 questions (95%) and open-ended MCQA alignment on 500 instances (80%).
Experimental setup
Datasets: MMLU HellaSwag ARC-Challenge ARC-Easy BoolQ SIQA PIQA AGIEval OpenBookQA CommonSenseQA RACE Google Trends
Benchmarks: MMLU HellaSwag ARC-Challenge ARC-Easy BoolQ SIQA PIQA AGIEval OpenBookQA CommonSenseQA RACE RWQ
Metrics: accuracy Elo rating alignment rate with human Kendall's tau rating difference
Baselines: AlpacaEval MT-Bench Chatbot Arena MCQA evaluation
Builds on: Elo rating system AlpacaEval MT-Bench
Closest papers by meaning
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.91)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.90)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
- Improving Automatic VQA Evaluation Using Large Language Models (AAAI 2024 · similarity 0.89)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.89)
- Listening to the Wise Few: Query–Key Alignment Unlocks Latent Correct Answers in Large Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.