Research map / Graph learning / Knowledge and temporal graphs
Dialogues Are Not Just Text: Modeling Cognition for Dialogue Coherence Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCGEval, a dialogue coherence evaluator that builds an AMR-based Dialogue Cognition Graph (with ConceptNet commonsense) for four cognitive abilities and fuses it with transformer text encoding via an information interaction enhancement module.
Problem
Existing automatic dialogue coherence metrics correlate weakly with human judgments because they ignore cognitive abilities (commonsense, core semantics, temporal order, role consistency) needed to detect dialogue errors.
Approach
Models four dialogue cognitive abilities in one unified AMR-based graph (DCG): utterance AMRs for core semantics, utterance-index and speaker-role nodes for temporal and role cognition, and ConceptNet embeddings for commonsense; graph reasoning output is then interactively fused with text representations (IIE) to predict a score.
Key results
- DCGEval reaches average correlation 0.513 on ConvAI2 versus 0.462 for ChatGPT and 0.501 for QuantiDCE (best prior).
- On EmpatheticDialogues DCGEval averages 0.393 versus 0.360 for QuantiDCE, a 3.3% gain.
- Removing the whole DCG graph drops average correlation by about 7.6% on both datasets (ConvAI2 0.461 vs 0.513 in ablation table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add commonsense cognition using ConceptNet node initialization (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.497 -> 0.513; 0.350 -> 0.393 | Helped |
| add core-semantic cognition with utterance AMR graphs (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.500 -> 0.513; 0.379 -> 0.393 | Helped |
| add temporal cognition (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.497 -> 0.513; 0.362 -> 0.393 | Helped |
| add role cognition (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.484 -> 0.513; 0.384 -> 0.393 | Helped |
| add temporal and role cognition together (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.480 -> 0.513; 0.360 -> 0.393 | Helped |
| add the DCG graph (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.461 -> 0.513; 0.293 -> 0.393 | Helped |
| use the IIE module instead of direct concatenation (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.466 -> 0.513; 0.318 -> 0.393 | Helped |
| use the IIE module instead of a two-layer MLP (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.473 -> 0.513; 0.348 -> 0.393 | Helped |
| use the full IIE module rather than hinter alone (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.495 -> 0.513; 0.358 -> 0.393 | Helped |
| use the full IIE module rather than xinter alone (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.508 -> 0.513; 0.346 -> 0.393 | Mixed |
| use the full IIE module rather than concatenated hinter and xinter (ablation) | DCGEval · ConvAI2; EmpatheticDialogues | Average correlation: 0.504 -> 0.513; 0.332 -> 0.393 | Helped |
Surprising findings
- Ablating the role cognition alone causes a smaller drop than ablating temporal and role together, and DynaEval performs very poorly despite modeling dialogue graphs.
- ChatGPT scores were unstable across two runs with the same configuration, with large variation on the same cases.
Limitations
- Depends on a pre-trained AMR parser, which can produce incorrect graphs (mostly on relation types).
- Evaluated on only two unseen daily dialogue datasets with human scores.
- ChatGPT baseline outputs are unstable across runs.
Experimental setup
Datasets: DailyDialog++ DailyDialogEVAL ConceptNet ConvAI2 EmpatheticDialogues
Benchmarks: ConvAI2 EmpatheticDialogues
Metrics: Pearson Spearman Kendall
Baselines: BLEU ROUGE METEOR BERTScore ADEM BERT-RUBER BLEURT QuantiDCE DynaEval GRADE ChatGPT
Builds on: GRADE DynaEval QuantiDCE BERT AMR parsing
Closest papers by meaning
- Dynamic Cognitive Planning for Cognitive-Functional Dialogue: A Case Study in Emotional Support Conversation (AAAI 2026 · similarity 0.85)
- Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (NeurIPS 2019 · similarity 0.85)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.84)
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.83)
- CogniVerse: Revolutionizing Multi-Modal Retrieval-Augmented Generation with Cognitive Reflection and Geometric Reasoning (CVPR 2026 · similarity 0.82)
- CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.