Research map / Large language models / Language, safety and interpretability
Code-switching Mediated Sentence-level Semantic Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a task-free sentence-level semantic invariance loss that aligns sentence representations across code-switched, English, and Chinese outputs in a unified conformer-transformer ASR/AST model, and reports gains on ASR, AST, and LLM code-switching generation.
Problem
Code-switched speech and text are handled poorly by monolingual-oriented NLP systems, and existing approaches do not explicitly model sentence-level semantic associations between different code-switched surface forms of the same meaning.
Approach
Treat different code-switched and monolingual renderings of the same sentence as semantically equivalent and add a loss (KL/MSE between sentence-level vectors from [CLS] or average pooling) that enforces semantic invariance across the ASR and two AST outputs in a multi-task decoder.
Key results
- On ASRU 2019 test, the [CLS]+MSE variant reaches 10.37 MER and 82.61 BLEUcl versus 11.01 MER and 78.98 for the multi-task baseline.
- On TED English-Chinese, the proposed method reaches 11.19 WER versus 13.38 for interactive decoding, and 22.11 BLEU for the best variant versus 21.68 for interactive decoding.
- For llama2-13B, the proportion of qualified generated code-switching text rises from 44% (no finetune) to 59% with the proposed loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add semantic invariance loss (ablation) | Proposed speech model · ASRU 2019 Dev | MER: 10.91 -> 10.55 | Helped |
| add semantic invariance loss (ablation) | Proposed speech model · ASRU 2019 Dev | BLEUcl: 80.37 -> 81.42 | Helped |
| add semantic invariance loss (ablation) | Proposed speech model · ASRU 2019 Test | MER: 10.77 -> 10.37 | Helped |
| add semantic invariance loss (ablation) | Proposed speech model · ASRU 2019 Test | BLEUcl: 80.81 -> 82.61 | Helped |
| add semantic invariance loss (ablation) | Proposed speech model · Fisher test | MER: 29.33 -> 28.72 | Helped |
| add semantic invariance loss (ablation) | Proposed speech model · Fisher test | BLEUci: 25.56 -> 27.02 | Helped |
| add semantic invariance loss (ablation) | Proposed speech model · TED English-Chinese test | MER: 12.32 -> 11.19 | Helped |
| add semantic invariance loss (ablation) | Proposed speech model · TED English-Chinese test | BLEUcl: 21.07 -> 21.50 | Helped |
| use [CLS] extraction with MSE rather than average pooling with MSE (ablation) | Proposed model · ASRU 2019 Test | MER: 10.51 -> 10.37 | Helped |
| use [CLS] extraction with MSE rather than average pooling with MSE (ablation) | Proposed model · ASRU 2019 Test | BLEUcl: 82.52 -> 82.61 | Helped |
| fine-tune with semantic invariant loss instead of supervised fine-tuning (ablation) | llama2-7B · LLM code-switching generation | qualification rate: 28% -> 33% | Helped |
| fine-tune with semantic invariant loss instead of supervised fine-tuning (ablation) | llama2-13B · LLM code-switching generation | qualification rate: 53% -> 59% | Helped |
Surprising findings
- Ablation shows the unified multi-task architecture still beats baselines even without the semantic invariance loss.
- Semantic invariance loss helps AST more than ASR in the ablation.
Limitations
- Semantic invariance is evaluated mainly on a small set of language pairs (Mandarin-English, English-Spanish data only in Fisher table) and tasks built on the same audio.
- LLM code-switching generation evaluated on only 100 samples with manual qualification judgments.
Experimental setup
Datasets: ASRU 2019 Mandarin-English code-switching challenge Fisher TED English-Chinese
Benchmarks: ASRU 2019 Fisher TED English-Chinese
Metrics: MER WER BLEU BLEUci BLEUcl qualification rate
Baselines: Pretrained model Multi-task model Interactive decoding (Liu et al. 2020) COSTT (Dong et al. 2021a) Transformer+pretrain (Liu et al. 2019) Multi-task+pretrain (Inaguma et al. 2019)
Builds on: Conformer CTC BERT multi-task learning
Closest papers by meaning
- SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Dataset (NeurIPS 2025 · similarity 0.86)
- Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems (AAAI 2020 · similarity 0.85)
- How Do Language Models Speak Languages? A Case Study on Unintended Code-Switching (ICML 2026 · similarity 0.84)
- DA-Net: A Disentangled and Adaptive Network for Multi-Source Cross-Lingual Transfer Learning (AAAI 2024 · similarity 0.83)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.82)
- Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.