Research map / Large language models / Retrieval and knowledge
Benchmarking Foundation Models with Language-Model-as-an-Examiner
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 as a reference-free examiner instead of lexical-overlap metrics | LMExamQA response evaluation · LMExamQA human annotations | Spearman ρ: BLEU 0.318; GPT-4 0.633 | Improved |
| Use GPT-4 as a reference-free examiner instead of lexical-overlap metrics | LMExamQA response evaluation · LMExamQA human annotations | Kendall τ: BLEU 0.241; GPT-4 0.554 | Improved |
| Use GPT-4 pairwise comparison instead of automatic text-similarity metrics | LMExamQA response evaluation · LMExamQA human pairwise comparisons | pairwise comparison accuracy: ROUGE-2 0.533; GPT-4 0.853 | Improved |
| Add five in-context examples | Models without SFT on LMExamQA · Memorization questions | relative performance: +17% | Improved |
| Add five in-context examples | Models without SFT on LMExamQA · Comprehension questions | relative performance: +123% | Improved |
| Add five in-context examples | Models without SFT on LMExamQA · Analysis questions | relative performance: +206% | Improved |
| Scale from LLaMA-13B to LLaMA-65B | LLaMA on LMExamQA · LMExamQA, all question categories | performance | Improved |
| Fine-tune LLaMA-13B on user-shared ChatGPT conversations to create Vicuna-13B | LLaMA-13B · LMExamQA, especially comprehension and analysis | performance | Improved |
| Use ChatGPT instead of Vicuna-13B as the response model | LMExamQA ranking · LMExamQA | win rate: 68% | Improved |
| Add follow-up questions after an initially correct response | Examinee models in multi-round QA · LMExamQA second-round QA | performance | Hurt |
| Combine multiple examiners' assessments by voting instead of relying on GPT-4 alone | Peer-examination · Style-controlled response comparisons | evaluation fairness | Improved |
| Use GPT-4 to evaluate responses to questions it did not formulate | GPT-4 evaluator on Natural Questions · Natural Questions | incorrect judgments: 18 out of 100 | Hurt |
Closest papers by meaning
- Reliable and Efficient Amortized Model-based Evaluation (ICML 2025 · similarity 0.91)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.90)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.89)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.89)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
