Research map / Large language models / Language, safety and interpretability
icsPLMs: Exploring Pre-trained Language Models in Intelligent Customer Service (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes icsPLMs, a StructBERT-based pre-trained language model adapted to intelligent customer service via attention-guided domain word masking and multi-task sentence-level pre-training, improving query matching and intent classification.
Problem
Generic pre-trained language models do not exploit domain-specific information in intelligent customer service text, limiting performance on intent and query matching tasks.
Approach
Automatically builds a domain lexicon from high-attention tokens in a fine-tuned intent classifier and uses it for an eMLM-style masking ratio (50% domain words), combined with multi-task sentence-level objectives (QQ matching, QA matching, and StructBERT generic task) during pre-training.
Key results
- icsPLM-MT reaches 88.50 on QM, 91.86 on IntC and 84.78 on ComIntC versus StructBERT's 87.50, 90.82 and 82.75.
- Largest gain is on ComIntC (+2.03 points over StructBERT); QM and IntC gain 1.00 and 1.04 points.
- Domain word masking alone (icsPLM) improves over StructBERT on all three tasks (e.g., 88.02 QM, 91.15 IntC, 83.16 ComIntC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add domain word masking and multi-task training | StructBert · QM | AUC: 87.50 -> 88.50 | Helped |
| Add domain word masking and multi-task training | StructBert · IntC | accuracy: 90.82 -> 91.86 | Helped |
| Add domain word masking and multi-task training | StructBert · ComIntC | accuracy: 82.75 -> 84.78 | Helped |
| Add domain word masking (ablation) | StructBert · QM | AUC: 87.50 -> 88.02 | Helped |
| Add domain word masking (ablation) | StructBert · IntC | accuracy: 90.82 -> 91.15 | Helped |
| Add domain word masking (ablation) | StructBert · ComIntC | accuracy: 82.75 -> 83.16 | Helped |
| Add multi-task training (ablation) | icsPLM · QM | AUC: 88.02 -> 88.50 | Helped |
| Add multi-task training (ablation) | icsPLM · IntC | accuracy: 91.15 -> 91.86 | Helped |
| Add multi-task training (ablation) | icsPLM · ComIntC | accuracy: 83.16 -> 84.78 | Helped |
Limitations
- Evaluated only on proprietary JIMI datasets in one Chinese customer service setting
- Gains are small (about 1-2 points) with no significance testing reported
- Only compared against StructBERT; no comparison with other domain-adapted models such as eMLM
Experimental setup
Datasets: JD Instant Messaging intelligence platform (JIMI) utterances JIMI query matching data JIMI intent classification data generic corpus
Benchmarks: QM (JIMI query matching) IntC (JIMI intent classification) ComIntC (JIMI complicated intent classification)
Metrics: AUC accuracy
Baselines: StructBERT
Builds on: StructBERT eMLM
Closest papers by meaning
- OWL: A Large Language Model for IT Operations (ICLR 2024 · similarity 0.80)
- EcomGPT: Instruction-Tuning Large Language Models with Chain-of-Task Tasks for E-commerce (AAAI 2024 · similarity 0.79)
- DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.78)
- Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges (AAAI 2025 · similarity 0.78)
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.78)
- Mix-Ecom: Towards Mixed-Type E-Commerce Dialogues with Complex Domain Rules (ICLR 2026 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.