Research map / Large language models / Language, safety and interpretability
LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAMPAT, an unsupervised multilingual paraphrase generator that fine-tunes mGPT with LoRA on corrupted monolingual WMT19 sentences and applies virtual adversarial training with embedding perturbations.
Problem
Multilingual paraphrase generation typically relies on parallel translation corpora, which are unavailable for many low-resource languages, and MT-based paraphrasers often produce outputs too similar to the input.
Approach
Train a paraphraser without parallel data by reconstructing sentences from stopword-removed, shuffled corruptions, and add LoRA-based parameter-efficient tuning plus virtual adversarial training (embedding-space perturbations with model-generated virtual labels, PGD and projected Newton steps) to encourage lexically diverse outputs.
Key results
- LAMPAT achieves the highest BERTScore on all 4 DAE-test languages (e.g., en 86.86 vs MMT 84.93) and the best BERT-iBLEU and ParaScore across the evaluated languages.
- On the input-reference set, LAMPAT reports higher BLEU than DAE and MMT on most of the 12 languages (e.g., en 23.07 vs 11.79 for MMT).
- In human evaluation averaged over 5 languages, LAMPAT scores 4.2 on semantic preservation versus 3.2 for MMT and lower lexical similarity (3.2 vs 4.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LAMPAT instead of MMT | Multilingual paraphrase generation · English, Spanish, and Russian test sets | ParaScore: 89.45 -> 92.95; 91.56 -> 92.96; 88.47 -> 91.92 | Helped |
| Use LAMPAT instead of MMT | Multilingual paraphrase generation · Chinese test set | ParaScore: 90.05 -> 90.64 | Helped |
| Use LAMPAT instead of MMT | Multilingual paraphrase generation · Input-only: French, Czech, Finnish, German, English, and Chinese | ParaScore: 89.95 -> 93.24; 90.10 -> 92.45; 81.15 -> 86.82; 89.47 -> 93.21; 91.49 -> 94.84; 90.35 -> 92.24 | Helped |
| Use LAMPAT instead of the strongest baseline | Multilingual paraphrase generation · Input-reference: 12 languages | ParaScore: See Table 3 | Helped |
| Use LAMPAT instead of MMT | Human-evaluated paraphrase generation · Five languages | Semantic preservation: 3.2 -> 4.2 | Helped |
| Use LAMPAT instead of MMT | Human-evaluated paraphrase generation · Five languages | Lexical similarity: 4.8 -> 3.2 | Helped |
| Use LAMPAT instead of MMT | Human-evaluated paraphrase generation · Five languages | Fluency: 4.8 -> 4.8 | No clear effect |
| Use LoRA instead of Prefix Tuning (ablation) | LAMPAT fine-tuning · 13 languages | ParaScore: 77.45±8.5 -> 90.67±2.5 | Helped |
| Use LoRA instead of Prompt Tuning (ablation) | LAMPAT fine-tuning · 13 languages | ParaScore: 82.25±7.9 -> 90.67±2.5 | Helped |
| Use LoRA instead of P-Tuning (ablation) | LAMPAT fine-tuning · 13 languages | ParaScore: 82.94±5.5 -> 90.67±2.5 | Helped |
| Use LoRA instead of Adapter (ablation) | LAMPAT fine-tuning · 13 languages | ParaScore: 89.56±3.2 -> 90.67±2.5 | Helped |
| Use LoRA instead of full fine-tuning (ablation) | LAMPAT fine-tuning · 13 languages | ParaScore: 89.46±1.5 -> 90.67±2.5 | Helped |
Surprising findings
- Unsupervised LAMPAT outperforms the supervised MT-based baseline MMT on most metrics, as claimed by the authors.
- LoRA outperforms full fine-tuning in the PEFT ablation (ParaScore 90.67 vs 89.46), and full fine-tuning has a lower std than some PEFT methods.
Limitations
- Struggles with idiomatic expressions according to the authors
- Evaluation covers only 13 languages, leaving out many low-resource languages
- Baseline comparisons rely on re-training MMT and DAE on WMT19 with limited human evaluation (200 triples per language)
Experimental setup
Datasets: WMT19 monolingual (training) WMT19 bilingual (MMT baseline) WMT19 validation (input-only eval) PAWS-X Opusparcus STAPLE
Benchmarks: DAE test set (en, es, zh, ru) input-only evaluation (fr, cs, fi, de, en, zh) input-reference evaluation (sv, fi, en, fr, de, ru, ja, zh, es, hu, pt, vi)
Metrics: BERTScore BLEU Self-BLEU Self-TER iBLEU BERT-iBLEU ParaScore
Baselines: MMT (multilingual machine translation paraphrasing) DAE (denoising auto-encoder)
Builds on: LoRA Virtual Adversarial Training mGPT Projected Newton Method YOPO
Closest papers by meaning
- PARABANK: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-Constrained Neural Machine Translation (AAAI 2019 · similarity 0.87)
- Paraphrase Diversification Using Counterfactual Debiasing (AAAI 2019 · similarity 0.86)
- Monolingual Transfer Learning via Bilingual Translators for Style-Sensitive Paraphrase Generation (AAAI 2020 · similarity 0.85)
- Pre-training via Paraphrasing (NeurIPS 2020 · similarity 0.85)
- Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text (NeurIPS 2025 · similarity 0.84)
- MT-Ranker: Reference-free machine translation evaluation by inter-system ranking (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.