Research map / Label-efficient and robust learning / Self-supervised and contrastive
Better Correlation and Robustness: A Distribution-Balanced Self-Supervised Learning Framework for Automatic Dialogue Evaluation
Peiwen Yuan, Xinglin Wang, Jiayi Shi, Bin Sun, Yiwei Li, Kan Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove whole training set reconstructing (TSR) (ablation) | BERT+BCR · DSTC10 overall | Spearman correlation: 0.261 -> 0.316 | Improved |
| Add pseudo label in TSR (ablation) | BERT+BCR · DSTC10 overall | Spearman correlation: 0.310 -> 0.316 | Improved |
| Add TSR on medium coherence samples (ablation) | BERT+BCR · DSTC10 medium coherence | Spearman correlation: 0.073 -> 0.131 | Improved |
| Add TSR on polarized coherence samples (ablation) | BERT+BCR · DSTC10 polarized coherence | Spearman correlation: 0.125 -> 0.163 | Improved |
| Apply BCR to baseline BERT | BERT-base · 17 datasets average | Spearman correlation: 0.323 -> 0.415 | Improved |
| Apply BCR to BERT vs prior SOTA methods | BERT+BCR · 17 datasets average | Spearman correlation: 0.373 -> 0.415 | Improved |
| Apply BCR to BERT on GDR | BERT+BCR · GDR | Spearman correlation: 0.197 -> 0.487 | Improved |
| Apply BCR to BERT on PE | BERT+BCR · PE | Spearman correlation: 0.612 -> 0.669 | Mixed |
| Apply BCR to GRADE and USR | GRADE, USR · DSTC10 | Spearman correlation and Diff: 0.171 -> 0.242 | Improved |
| Use DP loss instead of MAE, CE, MSE or Triplet loss (ablation) | BERT+BCR · DSTC10 | Spearman and robustness Diff | Improved |
| Train on Empathetic instead of DailyDialog | BERT+BCR · DSTC10 | Spearman correlation: 0.316 -> 0.191 | Hurt |
| BCR pre-training then supervised finetuning vs supervised only | BERT · FED split 7:3 | Spearman correlation: 0.240 -> 0.372 | Improved |
Closest papers by meaning
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.84)
- An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems (ICML 2024 · similarity 0.84)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.83)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.82)
- Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.82)
- Aligning Spoken Dialogue Models from User Interactions (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
