Research map / Large language models / Language, safety and interpretability
Diplomat: A Dialogue Dataset for Situated PragMATic Reasoning
Hengli Li, Song-Chun Zhu, Zilong Zheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide human-annotated rationales as input (ablation) | BART-base · DiPlomat CQA | accuracy: 20.2 -> 29.6 | Improved |
| provide human-annotated rationales as input (ablation) | T5-small · DiPlomat CQA | accuracy: 24.9 -> 34.1 | Improved |
| provide human-annotated rationales as input (ablation) | mT5-small · DiPlomat CQA | accuracy: 19.7 -> 29.8 | Improved |
| provide human-annotated rationales as input (ablation) | UnifiedQA-base · DiPlomat CQA | accuracy: 28.9 -> 38.8 | Improved |
| provide human-annotated rationales as input (ablation) | UnifiedQA-large · DiPlomat CQA | accuracy: 32.8 -> 42.4 | Improved |
| provide human-annotated rationales as input (ablation) | ChatGPT (pm) · DiPlomat CQA | accuracy: 40.6 -> 45.1 | Improved |
| provide human-annotated rationales as input (ablation) | ChatGPT (em) · DiPlomat CQA | accuracy: 1.0 -> 1.5 | No clear effect |
| add chain-of-thought prompt 'Let's think step by step' (ablation) | ChatGPT · DiPlomat zero-shot NLI | accuracy: 85.7 -> 63.8 | Hurt |
| give the full dialogue context (ChatGPT) instead of only the pragmatic turn (DeBERTa/T5) | NLI models · DiPlomat zero-shot NLI | accuracy: 45.3 -> 85.7 | Improved |
| require model to identify pragmatic turn then reason (C->PR) vs reasoning given the turn (CP->R) | BERT-base · DiPlomat | accuracy: 91.3 -> 50.2 | Hurt |
| scale RoBERTa from base to large | RoBERTa · DiPlomat CP->R | accuracy: 92.0 -> 60.8 | Hurt |
| scale RoBERTa from base to large | RoBERTa · DiPlomat C->PR | accuracy: 50.0 -> 0.0 | Hurt |
Closest papers by meaning
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems (NeurIPS 2025 · similarity 0.88)
- Bridging semantics and pragmatics in information-theoretic emergent communication (NeurIPS 2024 · similarity 0.88)
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.88)
- CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models (AAAI 2024 · similarity 0.87)
- HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.87)
- StateTree: Enhancing Long-term Dialogue Reasoning via Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
