Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Pre-training via Paraphrasing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train with retrieval+reconstruction (MARGE) instead of MLM (XLM-R) | multilingual pre-training · BUCC | F1: 66.0 -> 75.9 | Helped |
| MARGE vs XLM-R | multilingual pre-training · MLQA | F1: 71.6 -> 71.0 | No clear effect |
| MARGE vs XLM-R | multilingual pre-training · PAWS-X | accuracy: 86.4 -> 86.5 | No clear effect |
| MARGE vs mBART (document) | seq2seq pre-training · Document MT en-de | BLEU: 38.5 -> 39.2 | Helped |
| MARGE vs mBART (document) | seq2seq pre-training · Document MT zh-en | BLEU: 29.6 -> 28.4 | Hurt |
| train one model on all languages instead of one language | MARGE-NEWS · MLSum | ROUGE-L: 19.03 -> 19.09 | No clear effect |
| further pre-train on Wikipedia (MARGE vs MARGE-NEWS) (ablation) | MARGE-NEWS · MLSum | ROUGE-L: 19.09 -> 18.87 | Hurt |
Closest papers by meaning
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.87)
- In-Context Pretraining: Language Modeling Beyond Document Boundaries (ICLR 2024 · similarity 0.86)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.85)
- LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training (AAAI 2024 · similarity 0.85)
- Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.85)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.