Research map / Large language models / Language, safety and interpretability
Chinese Spelling Correction as Rephrasing Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReLM, which fine-tunes BERT to rephrase the whole sentence into mask slots instead of character-level tagging for Chinese spelling correction, achieving SOTA on ECSpell and LEMON and better multi-task transfer.
Problem
Chinese spelling correction models trained as character-to-character sequence tagging over-condition corrections on memorized error patterns, which hurts generalization, zero-shot transfer, and multi-task learning.
Approach
Replace character-level tagging with a rephrasing objective: concatenate the source sentence with a same-length sequence of mask characters and train a BERT-style masked LM to infill the whole corrected sentence, plus auxiliary masking of non-error characters to discourage learning alignment shortcuts.
Key results
- ReLM reaches F1 91.2, 82.4, and 83.6 on ECSpell LAW, MED, and ODW, beating the previous masked-fine-tuned SOTA by 10.6, 12.8, and 16.7 points.
- On zero-shot LEMON, ReLM raises the average F1 from 49.7 to 51.2 over prior tagging SOTA.
- In multi-task training, ReLM (prompt) keeps CSC F1 at 87.6 on LAW versus 62.0 for BERT-MFT tagging, and it lowers false positive rate to 6.8% average versus 13.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace character tagging with full-sentence rephrasing using fixed-length mask infilling | BERT on CSC · ECSpell LAW, MED, and ODW | F1: 80.6 -> 91.2; 69.6 -> 82.4; 66.9 -> 83.6 | Helped |
| Replace tagging with full-sentence rephrasing | BERT on zero-shot CSC · LEMON average | F1: 49.7 -> 51.2 | Helped |
| Replace tagging with full-sentence rephrasing | BERT on zero-shot CSC · LEMON GAM and NOV | F1: 33.3 -> 33.0; 38.6 -> 37.8 | Mixed |
| Apply rephrasing rather than tagging | GPT2 on CSC · ECSpell LAW | F1: 39.8 -> 71.2 | Helped |
| Mask only non-error characters rather than any characters for auxiliary MLM (ablation) | ReLM · ECSpell average | F1: 85.8 -> 88.1 | Helped |
| Set auxiliary MLM mask rate to 0% (ablation) | ReLM · ECSpell average | F1: 28.4 -> 57.8 | Hurt |
| Increase auxiliary MLM mask rate from 10% to 20% (ablation) | ReLM · ECSpell average | F1: 85.6 -> 87.7 | Helped |
| Increase auxiliary MLM mask rate from 20% to 30% (ablation) | ReLM · ECSpell average | F1: 87.7 -> 88.1 | Helped |
| Increase auxiliary MLM mask rate from 30% to 40% (ablation) | ReLM · ECSpell average | F1: 88.1 -> 86.3 | Hurt |
| Increase auxiliary MLM mask rate from 40% to 60% (ablation) | ReLM · ECSpell average | F1: 86.3 -> 82.4 | Hurt |
| Add learnable task prompts (ablation) | ReLM in multi-task CSC · ECSpell LAW, MED, and ODW | F1: 84.2 -> 87.6; 76.1 -> 80.8; 75.0 -> 78.0 | Helped |
| Jointly train CSC with other tasks | BERT tagging · ECSpell LAW, MED, and ODW | F1: 34.5; 15.1; 16.8 | Hurt |
Surprising findings
- ChatGPT with 10-shot prompting performs poorly on CSC (e.g., F1 26.0 on MED), which the authors attribute to lack of high-quality annotated CSC data on the web.
- Even a GPT2 model trained with the rephrasing objective outperforms BERT tagging by a large margin (F1 71.2 vs 39.8 on LAW).
Limitations
- Zero-shot LEMON training relies on 34 million synthesized sentence pairs, so gains depend on large synthetic data.
- Auto-regressive rephrasing (GPT2) underperforms ReLM, and the paper gives no in-depth study of why beyond fixed-length matching.
- Mask rate and strategy were tuned on the evaluation domains, with the authors noting results still depend on specific data.
Experimental setup
Datasets: ECSpell LEMON SIGHAN AFQMC TNEWS
Benchmarks: ECSpell LEMON SIGHAN AFQMC TNEWS
Metrics: precision recall F1 false positive rate
Baselines: BERT-Tagging BERT-MFT MDCSpell-MFT Soft-Masked-MFT CRASpell-MFT GPT2-Tagging Baichuan2-7B ChatGPT
Builds on: BERT masked fine-tuning MDCSpell prompt tuning
Closest papers by meaning
- CEC-Zero: Zero-Supervision Character Error Correction with Self-Generated Rewards (AAAI 2026 · similarity 0.89)
- ScholarGEC: Enhancing Controllability of Large Language Model for Chinese Academic Grammatical Error Correction (AAAI 2025 · similarity 0.86)
- Learning from Mistakes: Self-correct Adversarial Training for Chinese Unnatural Text Correction (AAAI 2025 · similarity 0.86)
- EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction (AAAI 2025 · similarity 0.82)
- RewriteLM: An Instruction-Tuned Large Language Model for Text Rewriting (AAAI 2024 · similarity 0.81)
- Learning from others' mistakes: Finetuning machine translation models with span-level error annotations (ICML 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.