Research map / Large language models / Language, safety and interpretability
From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Models to Pre-trained Machine Reader
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Retrofit RoBERTa with MRC-style pre-training | RoBERTa · MRQA 32-shot EQA | average F1: 45.6 -> 48.1 | Helped |
| Retrofit RoBERTa with MRC-style pre-training | RoBERTa · NER 4-shot | average F1: 39.4 -> 58.0 | Helped |
| Retrofit RoBERTa with MRC-style pre-training | RoBERTa · MRQA OOD EQA | average F1: +4.5 F1 on average | Helped |
| Retrofit RoBERTa with MRC-style pre-training | RoBERTa · EQA full-resource | average F1: 84.0 -> 84.9 | Helped |
| Retrofit RoBERTa with MRC-style pre-training | RoBERTa · NER full-resource | average F1: 80.8 -> 82.3 | Helped |
| Use MRC-formulated classification with MRC-style pre-training | RoBERTa · Sequence classification full-resource | average accuracy: 88.6 -> 89.1 | Helped |
| Add MRC-style pre-training to support rationale extraction | PMR on SST-2 · SST-2 test | reasonable extracted rationales: 74% | Helped |
| Select top 10 relevant contexts instead of random contexts (ablation) | MRC pre-training data construction · CoNLL | F1: 93.1 -> 93.1 | No clear effect |
| Select top 10 relevant contexts instead of random contexts (ablation) | MRC pre-training data construction · SQuAD | F1: 92.1 -> 92.2 | No clear effect |
| Select top 10 relevant contexts instead of random contexts (ablation) | MRC pre-training data construction · DREAM | accuracy: 70.7 -> 69.7 | No clear effect |
| Select contexts by top percentage rather than an absolute number (ablation) | MRC pre-training data construction · MRC pre-training | performance | Hurt |
| Use relevance-driven or diversity-driven pairing instead of random pairing (ablation) | MRC pre-training data construction with unanswerable examples · CoNLL, SQuAD, DREAM, and SST-2 | performance | No clear effect |
Closest papers by meaning
- Retrospective Reader for Machine Reading Comprehension (AAAI 2021 · similarity 0.88)
- PMRC: Prompt-Based Machine Reading Comprehension for Few-Shot Named Entity Recognition (AAAI 2024 · similarity 0.88)
- MPNet: Masked and Permuted Pre-training for Language Understanding (NeurIPS 2020 · similarity 0.86)
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More (ICML 2025 · similarity 0.85)
- Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis (AAAI 2025 · similarity 0.83)
- Progressive Inference: Explaining Decoder-Only Sequence Classification Models Using Intermediate Predictions (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.