Research map / Adversarial robustness and security / Attacks on text and graphs
Learning from Mistakes: Self-correct Adversarial Training for Chinese Unnatural Text Correction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LIMIT, a self-correct adversarial training framework for Chinese/English unnatural text correction that trains a T5-based generative corrector on its own beam-search mistakes with a ranking loss, plus a similarity-based decoding intervention.
Problem
Unnatural text correction (spelling errors and adversarial perturbations in Chinese and English) suffers from poor generalization due to exposure bias between training data and real-world inputs, and from over-correction of correct characters.
Approach
Generate negative and positive examples from the model's own top-K beam-search predictions, rank them by BLEU and similarity, and train with a ranking loss added to NLL; at inference, add a similarity term to the decoding score to keep outputs semantically consistent and reduce over-correction.
Key results
- On the Chinese correction benchmarks, LIMIT reaches 84.6 F1 on Perfect Pinyin and 81.2 F1 on Hybrid, versus 82.1 and 71.2 for the PROTECT-Finetune baseline.
- On perturbed Chinese NLU tasks, BERT+LIMIT improves over BERT by 0.6, 1.9, 3.1, 2.5 and 1.9 points on TNEWS, AFQMC, CMNLI, IFLYTEK and COLD (for example, COLD adversarial accuracy 82.4 vs 80.5).
- On adversarial CSL, LIMIT raises Rouge-1 from 52.9 (RobustGEC) to 58.9 and Rouge-L from 49.9 to 55.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LIMIT self-correct adversarial training and decoding intervention | Chinese unnatural text correction · Perfect Pinyin, Abbreviation Pinyin, Character Split, Hybrid, and Hybrid-v2 | F1: 84.6 vs. 82.1; 63.5 vs. 57.7; 93.2 vs. 91.8; 81.2 vs. 71.2; 66.8 vs. 59.6 | Helped |
| Add LIMIT as a defense module to BERT | BERT · Chinese NLU adversarial datasets | accuracy: 65.4 -> 66.0; 72.4 -> 72.4; 77.5 -> 79.1; 56.2 -> 59.7; 80.5 -> 82.4 | Helped |
| Add LIMIT as a defense module to BERT | BERT · AdvGLUE: Adv SST-2, Adv QQP, Adv MNLI-m, Adv RTE | accuracy: 32.3 -> 66.2; 50.8 -> 78.8; 32.6 -> 69.4; 37.0 -> 84.0 | Helped |
| Add LIMIT correction to the NLG system | Chinese NLG systems · ADGEN Adv, CSL Adv, LCSTS Adv | Rouge-L: 23.9 -> 24.9; 49.9 -> 55.5; 36.0 -> 36.1 | Helped |
| Add self-correct adversarial training (SC) (ablation) | Fine-tuning baseline · Hybrid | F1: 75.2 -> 75.6 | Helped |
| Add self-correct adversarial training (SC) (ablation) | Fine-tuning baseline · CMNLI | accuracy: 77.5 -> 78.8 | Helped |
| Add self-correct adversarial training (SC) (ablation) | Fine-tuning baseline · CSL | Rouge-L: 49.9 -> 55.4 | Helped |
| Add decoding intervention (DI) after SC (ablation) | Fine-tuning + SC · Hybrid | F1: 75.6 -> 81.2 | Helped |
| Add decoding intervention (DI) after SC (ablation) | Fine-tuning + SC · CMNLI | accuracy: 78.8 -> 79.1 | Helped |
| Add decoding intervention (DI) after SC (ablation) | Fine-tuning + SC · CSL | Rouge-L: 55.4 -> 55.5 | Helped |
| Use self-correct ranking loss instead of InfoNCE (ablation) | LIMIT · Hybrid | BLEU: 0.56 -> 0.57 | Helped |
| Use self-correct ranking loss instead of InfoNCE (ablation) | LIMIT · Hybrid | F1: 79.3% -> 81.2% | Helped |
Surprising findings
- Large language models such as GPT-3.5-Turbo with 10-shot prompting perform poorly on Chinese unnatural text correction (e.g., 1.2 F1 on Character Split).
- Adversarial training baselines (FreeLB, R3F, SMART, CreAT) lose performance on several clean Chinese NLU tasks relative to vanilla BERT.
Limitations
- Accuracy on abbreviated pinyin correction remains weak (F1 63.5 on Abb. Pinyin, with lower precision than PROTECT-Finetune on that subset).
- Evaluation relies on synthetic perturbations constructed by the authors, so real-world generalization is not directly measured.
Experimental setup
Datasets: PROTECT Chinese Wikipedia Comments2019 TNEWS AFQMC CMNLI IFLYTEK COLD advSST-2 advQQP advMNLI advRTE
Benchmarks: PROTECT Perfect Pinyin PROTECT Abbreviation Pinyin PROTECT Character Split PROTECT Hybrid Hybrid-v2 TNEWS AFQMC CMNLI IFLYTEK COLD AdvGLUE ADGEN
Metrics: Precision F1 Accuracy Rouge-1 Rouge-2 Rouge-L BLEU
Baselines: BERT SoftMasked MDCSpell PLOME MFT ReLM RobustGEC ATINTER PROTECT FreeLB SMART R3F
Builds on: PROTECT contrastive learning Match-Tuning FreeLB R3F
Closest papers by meaning
- Chinese Spelling Correction as Rephrasing Language Model (AAAI 2024 · similarity 0.86)
- A Theoretical Understanding of Self-Correction through In-context Alignment (NeurIPS 2024 · similarity 0.84)
- ScholarGEC: Enhancing Controllability of Large Language Model for Chinese Academic Grammatical Error Correction (AAAI 2025 · similarity 0.84)
- Enhancing NLU in Large Language Models Using Adversarial Noisy Instruction Tuning (AAAI 2025 · similarity 0.84)
- Training Language Models to Self-Correct via Reinforcement Learning (ICLR 2025 · similarity 0.84)
- Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.