Research map / Large language models / Retrieval and knowledge
RealTime QA: What's the Answer Right Now?
Jungo Kasai, Keisuke Sakaguchi, yoichi takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, Kentaro Inui
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| retrieve up-to-date news via Google custom search instead of a 2018 Wikipedia dump (DPR) | GPT-3 open-book · RealTime QA generation | exact match: 13.3 -> 34.6 | Improved |
| add open-book GCS retrieval to GPT-3 vs closed-book | GPT-3 · RealTime QA multiple choice | accuracy: 44.9 -> 66.5 | Improved |
| replace one choice with none of the above (NOTA) | GPT-3 with GCS · RealTime QA multiple choice | accuracy: 66.5 -> 58.4 | Hurt |
| insert dates in articles and questions (ablation) | open-book GPT-3 with GCS · first 6 weeks multiple choice | accuracy: 65.9 -> 69.3 | Improved |
| insert dates in articles and questions (ablation) | open-book GPT-3 with GCS · first 6 weeks NOTA | accuracy: 61.5 -> 59.8 | Mixed |
| insert date in question (ablation) | closed-book GPT-3 · first 6 weeks multiple choice | accuracy: 45.8 -> 39.7 | Hurt |
| use GPT-3 prompting instead of RAG for answer prediction with GCS retrieval | open-book · RealTime QA generation | exact match: 17.5 -> 34.6 | Improved |
| submit answers later than 24 hours after question announcement | open-book GPT-3 with GCS · RealTime QA | accuracy | Hurt |
Closest papers by meaning
- ChatQA: Surpassing GPT-4 on Conversational QA and RAG (NeurIPS 2024 · similarity 0.85)
- SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.84)
- TrustUQA: A Trustful Framework for Unified Structured Data Question Answering (AAAI 2025 · similarity 0.84)
- QGen Studio: An Adaptive Question-Answer Generation, Training and Evaluation Platform (AAAI 2025 · similarity 0.84)
- Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models (ICLR 2025 · similarity 0.84)
- CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
