Research map / Large language models / Retrieval and knowledge
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Drop function words so only context content words remain (ablation) | BERT-large on the 10 MRC datasets · 10 MRC datasets, relative average | relative performance: 89.2% of the original scores | No clear effect |
| Shuffle words within context sentences (ablation) | BERT-large on the 10 MRC datasets · 10 MRC datasets, relative average | relative performance: 78.5% of the original scores | No clear effect |
| Shuffle the order of context sentences (ablation) | BERT-large on the 10 MRC datasets · 10 MRC datasets, relative average | relative performance: 95.4% of the original scores | No clear effect |
| Replace context and question vocabulary with POS-based anonymized IDs (ablation) | BERT-large on SQuAD v1.1 · SQuAD v1.1 dev | F1: 91.5 -> 61.2 | Mixed |
| Replace vocabulary with POS-based anonymized IDs (ablation) | BERT-large on HotpotQA (distractor) · HotpotQA (distractor) dev | F1: 63.6 -> 16.8 | Hurt |
| Keep only the sentences most similar to the question (ablation) | BERT-large on HotpotQA (distractor) · HotpotQA (distractor) dev | F1: 63.6 -> 16.9 | Hurt |
| Remove all question words (ablation) | BERT-large on multiple-choice datasets · Multiple-choice datasets, relative average | relative performance: -23.9% | Mixed |
| Remove content words, leaving function words only (ablation) | BERT-large on MRC datasets · CoQA and SQuAD v1.1 dev | F1: 53.0; 17.4 | Mixed |
| Replace numerical expressions with random numbers (ablation) | BERT-large on HotpotQA (distractor) · HotpotQA (distractor) dev | F1: 63.6 -> 60.0 | No clear effect |
| Drop pronouns (ablation) | BERT-large on DuoRC · DuoRC dev | F1: 58.4 -> 57.0 | No clear effect |
| Train and evaluate using only content words (ablation) | BERT-large · 10 MRC datasets, relative upper bound average | relative performance: 92.5% | No clear effect |
| Train and evaluate with context words shuffled (ablation) | BERT-large · 10 MRC datasets, relative upper bound average | relative performance: 80.1% | No clear effect |
Closest papers by meaning
- Prerequisite Skills for Reading Comprehension: Multi-Perspective Analysis of MCTest Datasets and Systems (AAAI 2017 · similarity 0.89)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.87)
- Linguini: A benchmark for language-agnostic linguistic reasoning (NeurIPS 2025 · similarity 0.87)
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (NeurIPS 2024 · similarity 0.86)
- NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens (ICLR 2025 · similarity 0.86)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.