Research map / Large language models / Language, safety and interpretability
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Emotion+Infersent distillation | Hierarchical dialog models · Interactive human evaluation, Cornell and Reddit | overall interactive ratings: F(554,1)=9.016, p=.003 | Helped |
| Train on Reddit rather than Cornell | Hierarchical dialog models · Interactive human evaluation | overall interactive ratings: F(554,1)=30.796, p<.001 | Helped |
| Use the hybrid metric on self-play trajectories | Dialog model evaluation · Bot-level interactive human quality ratings | Pearson correlation: r>.7; previous metrics achieved r<.5 | Helped |
| Distill only emotion information instead of both emotion and Infersent (ablation) | EI dialog models · Cornell and Reddit automatic metrics | embedding metrics | Mixed |
| Distill only Infersent semantics instead of both emotion and Infersent (ablation) | EI dialog models · Cornell and Reddit automatic metrics | embedding metrics | Mixed |
| Provide emotion and Infersent embeddings as input without knowledge distillation (ablation) | EI dialog models · Cornell and Reddit automatic metrics | automatic metrics | Hurt |
| Optimize sentiment or semantic similarity directly in self-play | Self-play dialog trajectories · Self-play | diversity | Hurt |
Closest papers by meaning
- Towards Credible Human Evaluation of Open-Domain Dialog Systems Using Interactive Setup (AAAI 2023 · similarity 0.92)
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.87)
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (ICLR 2024 · similarity 0.86)
- IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering (NeurIPS 2024 · similarity 0.86)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.86)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.