Research map / Large language models / Retrieval and knowledge
CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CORECODE, a Chinese dyadic-dialogue commonsense dataset (76,787 domain-slot-value annotations over 19.7K dialogues) with six benchmark tasks, and shows that evaluated Chinese LLMs perform poorly and that LoRA fine-tuning gains are fragile under option perturbation.
Problem
Existing commonsense benchmarks are context-insensitive or cover few commonsense types, so LLMs' commonsense reasoning and conflict detection within open-domain dialogues is poorly evaluated, especially for Chinese.
Approach
Annotates fine-grained commonsense knowledge in dialogues using a standardized "domain: slot = value" ontology (9 domains, 37 slots across entity, event, social interaction), plus minimally and maximally modified conflict phrases, and derives six dialogue-level tasks from it.
Key results
- ChatGPT reaches only 0.275 accuracy on domain identification and 0.084 on slot identification under zero-shot evaluation.
- On the commonsense knowledge filling task, ChatGPT attains 0.896 accuracy, while the best open model ChatGLM2-6B reaches 0.818 accuracy on that task.
- Fine-tuning BLOOM-7.1B and ChatGLM-6B with LoRA improves performance, but accuracy collapses under option re-indicating and shuffling (e.g., BLOOM-7.1B drops to about 0.07 on slot identification after shuffling).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3-shot in-context examples instead of zero-shot (ablation) | ChatGLM-6B · CORECODE HARD, commonsense knowledge generation | F1: 0.147 -> 0.274 | Helped |
| Increase in-context examples from 3-shot to 5-shot (ablation) | ChatGLM-6B · CORECODE HARD, commonsense knowledge generation | F1: 0.274 -> 0.215 | Hurt |
| Use 3-shot in-context examples instead of zero-shot (ablation) | ChatGLM-6B · CORECODE HARD, commonsense conflict phrase detection | F1: 0.044 -> 0.060 | Helped |
| Increase in-context examples from 3-shot to 5-shot (ablation) | ChatGLM-6B · CORECODE HARD, commonsense conflict phrase detection | F1: 0.060 -> 0.052 | No clear effect |
| Fine-tune on CORECODE | BLOOM-7.1B and ChatGLM-6B · CORECODE; commonsense reasoning tasks | accuracy | Helped |
| Increase fine-tuning data size | BLOOM-7.1B and ChatGLM-6B · CORECODE, commonsense conflict phrase detection | F1 | Helped |
| Increase fine-tuning data size beyond approximately 4K instances | BLOOM-7.1B and ChatGLM-6B · CORECODE, commonsense knowledge generation | F1 | Mixed |
| Fine-tune BLOOM-7.1B rather than ChatGLM-6B | CORECODE fine-tuning · CORECODE, commonsense generation and conflict phrase detection | performance | Helped |
| Represent event subjects and predicates with speaker indicators A/B rather than x/y (ablation) | CORECODE benchmark evaluation · CORECODE EASY vs HARD, event-related tasks | F1 | Helped |
| Re-indicate answer options after fine-tuning (ablation) | BLOOM-7.1B and ChatGLM-6B · CORECODE selection tasks | accuracy | Hurt |
| Shuffle answer options after fine-tuning (ablation) | BLOOM-7.1B and ChatGLM-6B · CORECODE selection tasks | accuracy | Hurt |
| Both re-indicate and shuffle answer options after fine-tuning (ablation) | BLOOM-7.1B and ChatGLM-6B · CORECODE selection tasks | accuracy | Hurt |
Surprising findings
- BLOOM-7.1B after LoRA fine-tuning learns a shortcut mapping questions to option label 'b' and collapses to around 0.07 accuracy when options are shuffled.
- Performance under 5-shot prompting for ChatGLM-6B is lower than under 3-shot, attributed to long dialogue inputs.
Limitations
- Most evaluated models are small open Chinese LLMs; ChatGPT is only evaluated zero-shot on some tasks, and the paper notes LLM outputs often need filtering for automatic evaluation.
- Social interaction annotations are extremely rare (26 instances), so that dimension is essentially unevaluated.
- Annotator agreement among quality inspectors was moderate (Cohen's Kappa 59.4%).
Experimental setup
Datasets: NaturalConv DuLeMon ConceptNet CORECODE
Benchmarks: CORECODE
Metrics: accuracy F1 exact match BLEU METEOR ROUGE-L CIDEr ChatGPT score
Baselines: GLM-10B BLOOM-7.1B ChatGLM-6B ChatGLM2-6B BELLE-7B BLOOMZ-7.1B MOSS-SFT-16B Chinese-Alpaca-Plus-13B ChatGPT
Builds on: ConceptNet ATOMIC CICERO BLOOMZ LoRA
Closest papers by meaning
- Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures (NeurIPS 2026 · similarity 0.86)
- Improving Open-Domain Dialogue Response Generation with Multi-Source Multilingual Commonsense Knowledge (AAAI 2024 · similarity 0.86)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.86)
- StateTree: Enhancing Long-term Dialogue Reasoning via Reinforcement Learning (NeurIPS 2026 · similarity 0.85)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.85)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.