Research map / Large language models / Language, safety and interpretability
Evaluating Language Models in Realistic Conversational Contexts
Ilija Subasic, Andrew Rabinovich, Zhao Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn an ensemble (random forest) of automatic metrics instead of using single metrics | automatic evaluation metrics · UPHELD content accuracy | Pearson correlation: 0.58 -> 0.63 (best single message_embedding_cos_sim vs random forest) | Improved |
| learn an ensemble (random forest) of automatic metrics instead of using single metrics | automatic evaluation metrics · UPHELD style accuracy | Pearson correlation: 0.41 -> 0.61 | Improved |
| use random forest ensemble trained on UPHELD for out-of-domain data (ablation) | ensemble metric · LLM Arena and Topical-Chat | Pearson correlation: 0.63 -> 0.23 (LLM Arena content) | Hurt |
| use SVM or linear regression ensemble trained on UPHELD for out-of-domain data (ablation) | ensemble metric · LLM Arena and Topical-Chat | Pearson correlation: 0.54 (SVM, LLM Arena style) vs 0.51 best single | Improved |
| use LLM-as-a-judge instead of token or semantic metrics (ablation) | conversation evaluation · UPHELD | Pearson correlation: 0.26 (Likert) vs 0.58 (embedding cos sim) on content | Hurt |
| ask the LLM judge to explain before scoring (ablation) | llm_judge_yes_no · UPHELD content accuracy | Pearson correlation: 0.40 -> 0.23 | Hurt |
| ask the LLM judge to explain before scoring (ablation) | llm_judge_likert_1_5 · UPHELD content accuracy | Pearson correlation: 0.26 -> 0.28 | No clear effect |
| fine-tune Llama-3.1-70b on UPHELD held-out dialogues | Llama-3.1-70b · UPHELD content and style scores | human content/style score: approximately 40% increase in content and style accuracy | Improved |
| compare models on reference-free reasonableness | LLM candidates · UPHELD reasonableness | reasonableness score | No clear effect |
| restrict to higher human agreement bins (ablation) | LLM-as-a-judge · UPHELD | agreement with human consensus: 0.34 (plurality) -> 0.63 (full), Likert explain | Improved |
| use task-oriented dialogues instead of open-ended questions (ablation) | reference-full evaluation · semantic consistency of GPT-4o completions | semantic consistency (%): 74 -> 93 | Improved |
Closest papers by meaning
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.90)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.89)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.88)
- IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering (NeurIPS 2024 · similarity 0.87)
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (ICML 2024 · similarity 0.87)
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
