AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Evaluating Language Models in Realistic Conversational Contexts

Ilija Subasic, Andrew Rabinovich, Zhao Chen

ICML 2026 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
learn an ensemble (random forest) of automatic metrics instead of using single metricsautomatic evaluation metrics · UPHELD content accuracyPearson correlation: 0.58 -> 0.63 (best single message_embedding_cos_sim vs random forest)Improved
learn an ensemble (random forest) of automatic metrics instead of using single metricsautomatic evaluation metrics · UPHELD style accuracyPearson correlation: 0.41 -> 0.61Improved
use random forest ensemble trained on UPHELD for out-of-domain data (ablation)ensemble metric · LLM Arena and Topical-ChatPearson correlation: 0.63 -> 0.23 (LLM Arena content)Hurt
use SVM or linear regression ensemble trained on UPHELD for out-of-domain data (ablation)ensemble metric · LLM Arena and Topical-ChatPearson correlation: 0.54 (SVM, LLM Arena style) vs 0.51 best singleImproved
use LLM-as-a-judge instead of token or semantic metrics (ablation)conversation evaluation · UPHELDPearson correlation: 0.26 (Likert) vs 0.58 (embedding cos sim) on contentHurt
ask the LLM judge to explain before scoring (ablation)llm_judge_yes_no · UPHELD content accuracyPearson correlation: 0.40 -> 0.23Hurt
ask the LLM judge to explain before scoring (ablation)llm_judge_likert_1_5 · UPHELD content accuracyPearson correlation: 0.26 -> 0.28No clear effect
fine-tune Llama-3.1-70b on UPHELD held-out dialoguesLlama-3.1-70b · UPHELD content and style scoreshuman content/style score: approximately 40% increase in content and style accuracyImproved
compare models on reference-free reasonablenessLLM candidates · UPHELD reasonablenessreasonableness scoreNo clear effect
restrict to higher human agreement bins (ablation)LLM-as-a-judge · UPHELDagreement with human consensus: 0.34 (plurality) -> 0.63 (full), Likert explainImproved
use task-oriented dialogues instead of open-ended questions (ablation)reference-full evaluation · semantic consistency of GPT-4o completionssemantic consistency (%): 74 -> 93Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy