Research map / Large language models / Language, safety and interpretability
Towards Credible Human Evaluation of Open-Domain Dialog Systems Using Interactive Setup
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SOBA instead of SATA for pairwise comparisons | Interactive pairwise evaluation · GPT2XL-BST vs GPT2XL-TCS | turns required: 196 vs 667 | Helped |
| Use SOBA instead of SATA for pairwise comparisons | Interactive pairwise evaluation · GPT2XL-BST vs GPT2M-BST | win-rate difference sensitivity: SOBA: 61%; SATA-Random: 57%; SATA-User: 57% | Helped |
| Use SATA rather than SOBA for similarly performing model pairs | Interactive pairwise evaluation · GPT2XL-BST vs GPT2XL-WoW | consistency of model comparison: 62%; 57%; 54% | Helped |
| Add the SATA option to select none of the responses | Interactive per-turn dialog evaluation · Pairwise and multi-model evaluations | failure and tie measurement | Helped |
| Use SATA-User rather than SATA-Random fallback | Interactive per-turn dialog evaluation · Pairwise and multi-model evaluations | win-rate consistency: SATA-Random and SATA-User generate highly consistent and equally sensitive results | No clear effect |
| Evaluate four candidates together instead of two | Interactive multi-model evaluation · Four GPT2 models | per-model win-rate: roughly centered at 50% -> roughly centered at 25% | Hurt |
| Use MM-SATA rather than MM-SOBA | Interactive multi-model evaluation · Four GPT2 models | pairwise equality testing | Helped |
| Use single-model evaluation rather than pairwise or multi-model evaluation | Interactive evaluation of GPT2 models · Single-model evaluation | sensitivity to model differences: 84%-94% | Hurt |
| Increase GPT2 model size from GPT2M to GPT2XL | BST fine-tuned GPT2 · Pairwise and multi-model evaluations | win-rate: GPT2XL-BST: 61% vs GPT2M-BST: 39%; 57% vs 45% | Helped |
| Fine-tune GPT2XL on BST rather than TCS | GPT2XL · Pairwise and multi-model evaluations | win-rate: 62% vs 38%; 58% vs 48% | Helped |
| Fine-tune GPT2XL on BST rather than WoW | GPT2XL · Pairwise and multi-model evaluations | win-rate: 62% vs 38%; 57% vs 51%; 54% vs 48% | No clear effect |
| Estimate sample size before running the evaluation | Interactive human evaluation · Pairwise SOBA vs SATA | required turns: 196; 667 | Helped |
Closest papers by meaning
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.88)
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics (ICLR 2025 · similarity 0.87)
- IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering (NeurIPS 2024 · similarity 0.86)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.86)
- ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Ablation Capability for Large Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.