Research map / Adversarial robustness and security / Attacks on text and graphs
Re-Evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace reference responses with machine-generated responses | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 2.64 | No clear effect |
| Use the context itself as the response | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 3.03 | No clear effect |
| Swap the reference response and response being evaluated | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 2.6 | No clear effect |
| Remove punctuation from otherwise high-rated responses | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 2.85 | No clear effect |
| Remove the 25 most common stopwords | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 2.80 | No clear effect |
| Remove NLTK stopwords | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 2.69 | No clear effect |
| Replace most words with synonyms while preserving meaning | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 2.83 | No clear effect |
| Jumble words in the response | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 2.73 | No clear effect |
| Reverse response word order | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 2.75 | No clear effect |
| Randomly repeat words in the response | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 2.70 | No clear effect |
| Use “fantastic! how are you?” regardless of context | ADEM · MSR Social Media Conversation Corpus | ADEM mean score: 2.75 -> 3.18 | Helped |
| Search the embedding database for the highest-scoring response | ADEM · Twitter response embedding search | ADEM mean score: 2.75 -> 3.93 | Helped |
Closest papers by meaning
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.89)
- Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems (AAAI 2020 · similarity 0.87)
- RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems (AAAI 2018 · similarity 0.87)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.84)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.84)
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.