AI Paper Atlas

Research map / Large language models / Retrieval and knowledge

Benchmarking Foundation Models with Language-Model-as-an-Examiner

Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li and 1 more

NeurIPS 2023 · Large language models › Retrieval and knowledge · approach: Explanatory and multiple-choice QA benchmarks

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use GPT-4 as a reference-free examiner instead of lexical-overlap metricsLMExamQA response evaluation · LMExamQA human annotationsSpearman ρ: BLEU 0.318; GPT-4 0.633Improved
Use GPT-4 as a reference-free examiner instead of lexical-overlap metricsLMExamQA response evaluation · LMExamQA human annotationsKendall τ: BLEU 0.241; GPT-4 0.554Improved
Use GPT-4 pairwise comparison instead of automatic text-similarity metricsLMExamQA response evaluation · LMExamQA human pairwise comparisonspairwise comparison accuracy: ROUGE-2 0.533; GPT-4 0.853Improved
Add five in-context examplesModels without SFT on LMExamQA · Memorization questionsrelative performance: +17%Improved
Add five in-context examplesModels without SFT on LMExamQA · Comprehension questionsrelative performance: +123%Improved
Add five in-context examplesModels without SFT on LMExamQA · Analysis questionsrelative performance: +206%Improved
Scale from LLaMA-13B to LLaMA-65BLLaMA on LMExamQA · LMExamQA, all question categoriesperformanceImproved
Fine-tune LLaMA-13B on user-shared ChatGPT conversations to create Vicuna-13BLLaMA-13B · LMExamQA, especially comprehension and analysisperformanceImproved
Use ChatGPT instead of Vicuna-13B as the response modelLMExamQA ranking · LMExamQAwin rate: 68%Improved
Add follow-up questions after an initially correct responseExaminee models in multi-round QA · LMExamQA second-round QAperformanceHurt
Combine multiple examiners' assessments by voting instead of relying on GPT-4 alonePeer-examination · Style-controlled response comparisonsevaluation fairnessImproved
Use GPT-4 to evaluate responses to questions it did not formulateGPT-4 evaluator on Natural Questions · Natural Questionsincorrect judgments: 18 out of 100Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy