Research map / Large language models / Retrieval and knowledge
Explanatory and multiple-choice QA benchmarks
Create datasets and analyses for causal, structured, and multiple-choice question answering.
152 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Retrieval and knowledge, keyword cluster “question · answering · editing”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
38 papers in 2023–24 and 49 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 16.7% to 7.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions (AAAI 2016 · 145 citations)
- 2017 · Building Task-Oriented Dialogue Systems for Online Shopping (AAAI 2017 · 168 citations)
- 2018 · A Knowledge-Grounded Neural Conversation Model (AAAI 2018 · 435 citations)
- 2019 · Wizard of Wikipedia: Knowledge-Powered Conversational Agents (ICLR 2019 · 623 citations)
- 2020 · PIQA: Reasoning about Physical Commonsense in Natural Language (AAAI 2020 · 659 citations)
- 2021 · Retrospective Reader for Machine Reading Comprehension (AAAI 2021 · 191 citations)
- 2022 · ISEEQ: Information Seeking Question Generation Using Dynamic Meta-Information Retrieval and Knowledge Graphs (AAAI 2022 · 52 citations)
- 2023 · Generate rather than Retrieve: Large Language Models are Strong Context Generators (ICLR 2023 · 85 citations)
- 2024 · FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering (AAAI 2024 · 77 citations)
- 2025 · Citations and Trust in LLM Generated Responses (AAAI 2025 · 22 citations)
- 2026 · CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering (ICLR 2026 · 3 citations)
Earliest papers here
- Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions (AAAI 2016 · 145 citations)
- Addressing a Question Answering Challenge by Combining Statistical Methods with Inductive Rule Learning and Reasoning (AAAI 2016 · 90 citations)
- Dialog-based Language Learning (NeurIPS 2016 · 82 citations)
What has been tried
113 papers here report 1,137 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 7%, hurt 18%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add execution-guided self-training (EGST) | GrailQA, 25 shots · F1 | 57.7 -> 69.4; 10.3 | Helped | FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering (AAAI 2024 · 77 citations) |
| add demonstrations, using the full set rather than 1 demonstration | Ablation validation settings · Spearman correlation | 59.07 -> 66.74 | Helped | Improving Automatic VQA Evaluation Using Large Language Models (AAAI 2024 · 36 citations) |
| Add Q&A and course-document retrieval context to the engineered LLM prompt | DATA 100 stand-alone questions · Factuality | 1.860±0.80 -> 2.040±0.80 | Helped | RetLLM-E: Retrieval-Prompt Strategy for Question-Answering on Student Discussion Forums (AAAI 2024 · 15 citations) |
| Add rationale generation to Numerical, Binary, or Scale abstention | iMedQA · accuracy | Helped | MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning (NeurIPS 2024 · 15 citations) | |
| add positive-sample cross-entropy loss | KdConv (music) · BLEU-1 | 36.59 -> 39.39 | Helped | Improving the Robustness of Knowledge-Grounded Dialogue via Contrastive Learning (AAAI 2024 · 11 citations) |
| add the Combine Information step | CICERO · F1 | 88.39(-1.13) -> 89.52 | Helped | Reverse Multi-Choice Dialogue Commonsense Inference with Graph-of-Thought (AAAI 2024 · 9 citations) |
| Add CoG stage 1 | SQA-I test · accuracy | 73.13 -> 74.92; 1.79 ↑ | Helped | CoG-DQA: Chain-of-Guiding Learning with Large Language Models for Diagram Question Answering (CVPR 2024 · 7 citations) |
| reduce the number of turns from 3 to 1 | Evaluated tasks · judge accuracy | 3 to 1 | No clear effect | On scalable oversight with weak LLMs judging strong LLMs (NeurIPS 2024 · 6 citations) |
| Add in-cluster rank penalization | MMK-DailyDialog multilingual · Mean | 12.40 -> 12.59 | Helped | Improving Open-Domain Dialogue Response Generation with Multi-Source Multilingual Commonsense Knowledge (AAAI 2024 · 5 citations) |
| Add faithfulness and reward filtering to WiFS | WiFS human assessment · question relevance | 67% -> 92% | Helped | Achieving Human Parity in Content-Grounded Datasets Generation (ICLR 2024 · 4 citations) |
| Use 3-shot in-context examples instead of zero-shot | CORECODE HARD, commonsense knowledge generation · F1 | 0.147 -> 0.274 | Helped | CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models (AAAI 2024 · 3 citations) |
| Add MRFS verification | QA benchmark suite · GPT-EM accuracy | 66.9 -> 70.7 (Web); 56.8 -> 57.4 (DATE); 95.7 -> 98.6 (GK); 81.5 -> 83.1 (Social); 65.0 -> 67.3 (Truth); 75.2 -> 79.2 (S | Helped | Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models (ICLR 2025 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 26 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- HotpotQA · 3 of 26 papers
- WebQSP · 2 of 26 papers
- MedQA · 2 of 26 papers
- TriviaQA · 2 of 26 papers
- PopQA · 2 of 26 papers
Baselines
- gpt-4o · 3 of 26 papers
- llama-3-70b · 3 of 26 papers
- GPT4 · 2 of 26 papers
- ToG · 2 of 26 papers
- ReAct · 2 of 26 papers
- gpt-3.5-turbo · 2 of 26 papers
- BM25 · 2 of 26 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 87 papers; from OpenAlex and the papers' affiliations).
- Amazon · 4 papers
- East China Normal University · 3 papers
- Tsinghua University · 3 papers
- Korea University · 3 papers
- University College London · 3 papers
- Fudan University · 3 papers
- IBM · 3 papers
- Peking University · 3 papers
- Google · 3 papers
- University of California, Los Angeles · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Knowledge graph reasoning with LLMs (174 papers)
- Knowledge editing in LLMs (170 papers)
- Knowledge-augmented question answering (134 papers)
- Domain-specific QA datasets (79 papers)
Elsewhere in the topic
- RAG robustness and evaluation (164 papers)
- Retrieval-augmented reasoning (130 papers)
- Long-context and personalized retrieval (61 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).