Research map / Large language models / Language, safety and interpretability
RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine referenced and unreferenced scores with arithmetic averaging | RUBER · Retrieval (Top-1) | Pearson correlation: 0.4278 -> 0.4594 | Helped |
| Combine referenced and unreferenced scores with geometric averaging | RUBER · Retrieval (Top-1) | Spearman correlation: 0.4338 -> 0.4771 | Helped |
| Use the max operator instead of min or averaging (ablation) | RUBER · Retrieval (Top-1) | Pearson correlation: 0.4594 -> 0.3263 | Hurt |
| Use the min operator instead of averaging (ablation) | RUBER · Retrieval (Top-1) | Pearson correlation: 0.4594 -> 0.4428 | No clear effect |
| Use geometric averaging rather than arithmetic averaging (ablation) | RUBER · Seq2Seq with attention | Pearson correlation: 0.4509 -> 0.4523 | No clear effect |
| Use the max operator rather than geometric averaging (ablation) | RUBER · Seq2Seq with attention | Pearson correlation: 0.4523 -> 0.3868 | Hurt |
| Add the unreferenced NN scorer to the referenced vector-pooling score | RUBER · Seq2Seq with attention | Pearson correlation: 0.3434 -> 0.4137 | Helped |
| Use the NN scorer instead of vector pooling for query-reply relevance (ablation) | Unreferenced metric · Seq2Seq with attention | Pearson correlation: 0.3808 -> 0.4137 | Helped |
| Use embedding-based vector pooling instead of word-overlap Rouge | Referenced metric · Seq2Seq with attention | Pearson correlation: 0.1747 -> 0.3434 | Helped |
| Transfer RUBER trained on Douban directly to Baidu Tieba | RUBER · Baidu Tieba transfer, Seq2Seq with attention | Pearson correlation: 0.2729 -> 0.3885 | Helped |
Closest papers by meaning
- Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (NeurIPS 2019 · similarity 0.92)
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.89)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.88)
- ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models (ICLR 2025 · similarity 0.86)
- DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English (NeurIPS 2026 · similarity 0.86)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.