Research map / Large language models / Instruction and fine-tuning
Evaluating, Synthesizing, and Enhancing for Customer Support Conversation
Jie Zhu, Huaixia Dou, Junhui Li, Lifan Guo, Feng Chen, Chi Zhang, Fang Kong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on RoleCS | LLaMA3.1-Instruct 8B · CSConv (reference context) | strategy accuracy: 17.16 -> 42.15 | Improved |
| fine-tune on RoleCS | Qwen2.5-Instruct 7B · CSConv (reference context) | strategy accuracy: 18.78 -> 42.96 | Improved |
| fine-tune on RoleCS | Qwen2.5-Instruct 72B · CSConv (reference context) | strategy accuracy: 37.22 -> 43.29 | Improved |
| fine-tune on RoleCS | Qwen2.5-Instruct 7B · CSConv (generated context) | strategy accuracy: 19.39 -> 30.52 | Improved |
| fine-tune on RoleCS | Qwen2.5-Instruct 72B · CSConv (generated context) | strategy accuracy: 30.93 -> 36.02 | Improved |
| use role-playing instead of in-context generation for synthetic data (ablation) | Qwen2.5-7B SFT · CSConv | strategy accuracy: 33.09 -> 36.17 | Improved |
| add Supporter Assistant agent to role-playing (ablation) | Qwen2.5-7B SFT · CSConv | strategy accuracy: 36.17 -> 42.96 | Improved |
| increase synthetic data beyond first 3K dialogues (ablation) | Qwen2.5-7B SFT · CSConv | BLEU-4 / ROUGE-L / BERTScore | No clear effect |
| predict strategy before responding vs none (ablation) | Qwen2.5-72B-Instruct · CSConv | BLEU-4: 3.11 -> 3.23 | Improved |
| provide oracle strategy instead of predicted (ablation) | Qwen2.5-72B-Instruct · CSConv | BLEU-4: 3.23 -> 3.80 | Improved |
| rewrite real conversations with LLM to use strategies | CSConv · strategy use rate | strategy use: 55.28% -> 97.82% | Improved |
| evaluate with model-generated history instead of gold history | all LLMs · CSConv | strategy accuracy | Hurt |
Closest papers by meaning
- SocialSim: Towards Socialized Simulation of Emotional Support Conversation (AAAI 2025 · similarity 0.82)
- A Framework for Mining Speech-to-Text Transcripts of the Customer for Automated Problem Remediation (AAAI 2024 · similarity 0.81)
- icsPLMs: Exploring Pre-trained Language Models in Intelligent Customer Service (Student Abstract) (AAAI 2024 · similarity 0.80)
- PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios (NeurIPS 2026 · similarity 0.80)
- AGILE: A Novel Reinforcement Learning Framework of LLM Agents (NeurIPS 2024 · similarity 0.80)
- Cooper: Coordinating Specialized Agents towards a Complex Dialogue Goal (AAAI 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
