Research map / Large language models / Retrieval and knowledge
Evaluating Open-QA Evaluation
Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, Yue Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Ask the evaluator to give reasons for its judgments (ablation) | GPT-3.5 evaluator · EVOUNA NQ-BingChat | accuracy/F1: 69.5/77.2 -> 55.6/62.2 | Hurt |
| Ask the evaluator to give reasons for its judgments (ablation) | GPT-3.5 evaluator · EVOUNA NQ-ChatGPT4 | accuracy/F1: 80.9/86.9 -> 64.3/71.2 | Hurt |
| Tell the evaluator to ignore background information (ablation) | GPT-3.5 evaluator · EVOUNA NQ-BingChat | accuracy/F1: 69.5/77.2 -> 65.7/73.4 | Hurt |
| Tell the evaluator to ignore background information (ablation) | GPT-3.5 evaluator · EVOUNA NQ-FiD | accuracy/F1: 93.6/95.3 -> 93.9/95.5 | No clear effect |
| Add chain-of-thought prompting (ablation) | GPT-3.5 evaluator · EVOUNA NQ-BingChat | accuracy/F1: 69.5/77.2 -> 80.4/87.1 | Improved |
| Add chain-of-thought prompting (ablation) | GPT-3.5 evaluator · EVOUNA NQ-ChatGPT4 | accuracy/F1: 80.9/86.9 -> 86.0/91.2 | Improved |
| Add chain-of-thought prompting (ablation) | GPT-3.5 evaluator · EVOUNA NQ-FiD | accuracy/F1: 93.6/95.3 -> 90.2/92.9 | Hurt |
| Add in-context learning examples (ablation) | GPT-3.5 evaluator · EVOUNA NQ-BingChat | accuracy/F1: 69.5/77.2 -> 75.3/82.5 | Improved |
| Add in-context learning examples (ablation) | GPT-3.5 evaluator · EVOUNA NQ-ChatGPT4 | accuracy/F1: 80.9/86.9 -> 83.2/88.6 | Improved |
| Add in-context learning examples (ablation) | GPT-3.5 evaluator · EVOUNA NQ-FiD | accuracy/F1: 93.6/95.3 -> 93.1/94.9 | No clear effect |
| Exclude the NQ-BingChat subset from evaluation (ablation) | GPT-3.5 evaluator · EVOUNA-NQ | false negative proportion: 14.3 -> 9.9 | Improved |
| Use lexical matching instead of human judgment to score QA models | Open-QA evaluation · NaturalQuestions ChatGPT-4 | accuracy: 78.8 -> 61.8 | Hurt |
Closest papers by meaning
- IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering (NeurIPS 2024 · similarity 0.91)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.90)
- Improving Automatic VQA Evaluation Using Large Language Models (AAAI 2024 · similarity 0.89)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.88)
- SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.88)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
