Research map / Large language models / AI and society
An Oral Exam for Measuring a Dialog System’s Capabilities
David Cohen, Ian Lane
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compare Human vs Google Now | oral exam · Dialog capability oral exam | total performance: 1.04 -> 3.63 | Improved |
| compare Google Now vs Cleverbot | oral exam · Dialog capability oral exam | total performance: 0.31 -> 1.04 | Improved |
| compare Google Now vs Cleverbot on Physical Object Ontology | oral exam · Dialog capability oral exam | performance c1: 0.17 -> 0.39 | Improved |
| compare Google Now vs Cleverbot on breadth | oral exam · Dialog capability oral exam | breadth: 0.71 -> 0.91 | No clear effect |
| compare Google Now vs Cleverbot on flexibility c1 | oral exam · Dialog capability oral exam | flexibility c1: 0.54 -> 0.48 | No clear effect |
| compare Google Now vs Cleverbot on flexibility c2 | oral exam · Dialog capability oral exam | flexibility c2: 0 -> 0.59 | No clear effect |
| compare Human vs Google Now on flexibility | oral exam · Dialog capability oral exam | flexibility c4: 0.65 -> 1 | Improved |
| compare Human vs Google Now on breadth | oral exam · Dialog capability oral exam | breadth: 0.91 -> 1 | Improved |
| Google Now specialized Get Definition vs similar questions | Google Now · Dialog capability oral exam | capability performance | Mixed |
| use groups of three judges for entropy metrics | breadth and flexibility · Dialog capability oral exam | reliability | Mixed |
Closest papers by meaning
- Towards Credible Human Evaluation of Open-Domain Dialog Systems Using Interactive Setup (AAAI 2023 · similarity 0.86)
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.84)
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics (ICLR 2025 · similarity 0.83)
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs (NeurIPS 2024 · similarity 0.82)
- An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems (ICML 2024 · similarity 0.82)
- Dynamic Evaluation of Large Language Models by Meta Probing Agents (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
