Research map / Label-efficient and robust learning / Test-time adaptation
The Effect of Natural Distribution Shift on Question Answering Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate SQuAD-trained models on newly collected Wikipedia examples from the same domain | SQuAD QA models · New Wikipedia | F1: 3.8 F1 points | No clear effect |
| Evaluate SQuAD-trained models on New York Times text | SQuAD QA models · New York Times | F1: 3.8 F1 points | Hurt |
| Evaluate SQuAD-trained models on Reddit text | SQuAD QA models · Reddit | F1: 14.0 F1 points | Hurt |
| Evaluate SQuAD-trained models on Amazon reviews | SQuAD QA models · Amazon | F1: 17.4 F1 points | Hurt |
| Train with additional question-answering datasets beyond SQuAD | MRQA models · Reddit and Amazon | F1 | Mixed |
| Manually filter malformed or invalid examples (ablation) | Models evaluated on the New Wikipedia and Amazon curated subsets · New Wikipedia and Amazon | F1: 0.86 F1 points; 0.09 F1 points | Mixed |
| Reweight for answer-category distribution differences (ablation) | SQuAD QA models · Reddit | F1: 2-3 F1 points; 14.0 F1 points | No clear effect |
| Reweight for syntactic-divergence distribution differences (ablation) | SQuAD QA models · Reddit | F1: 1.9 F1 points; 14.0 F1 points | No clear effect |
| Reweight for reasoning-category distribution differences (ablation) | SQuAD QA models · New test sets | F1 | No clear effect |
Closest papers by meaning
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.86)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.86)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.85)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.85)
- Representation Consistency for Accurate and Coherent LLM Answer Aggregation (NeurIPS 2025 · similarity 0.85)
- QGen Studio: An Adaptive Question-Answer Generation, Training and Evaluation Platform (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.