Research map / Vision-language and multimodal / Visual question answering
Large Language Models Struggle to Learn Long-Tail Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase relevant pre-training document count from 10^1 to 10^4 | BLOOM-176B · TriviaQA | Exact Match accuracy: 25% to above 55% | Helped |
| Increase model size from BLOOM-560M to BLOOM-176B | BLOOM · TriviaQA questions with more than 10^5 relevant documents | Exact Match accuracy: over 4× higher accuracy | Helped |
| Remove relevant pre-training documents for sampled questions (ablation) | 4.8B-parameter LM trained on C4 · TriviaQA questions whose relevant documents were removed | Exact Match accuracy | Hurt |
| Count documents containing both question and answer entities instead of only one entity (ablation) | Document-count heuristic · QA examples with fewer than 5 question-answer entity co-occurrences | correlation with QA accuracy | Helped |
| Use question-entity-only document counts instead of question-answer co-occurrences (ablation) | Document-count heuristic · QA examples with fewer than 5 question-answer entity co-occurrences | correlation with QA accuracy | Hurt |
| Use answer-entity-only document counts instead of question-answer co-occurrences (ablation) | Document-count heuristic · QA examples with fewer than 5 question-answer entity co-occurrences | correlation with QA accuracy | Hurt |
| Provide a gold Wikipedia paragraph as context | GPT-Neo · Natural Questions, especially questions with few relevant pre-training documents | Exact Match accuracy | Helped |
| Provide the top 3 BM25-retrieved paragraphs as context | GPT-Neo · Natural Questions, across relevant-document-count ranges | Exact Match accuracy | Helped |
| Use oracle-retrieved context rather than closed-book answering | GPT-Neo · Natural Questions, accuracy by relevant pre-training document count | Exact Match accuracy trend | Mixed |
| Increase dataset size by a moderate factor, such as 5x | Pre-training data · Long-tail QA | QA accuracy: 5x | No clear effect |
| Scale model size | Language models · Long-tail QA | Exact Match accuracy | Helped |
| Evaluate human annotators rather than closed-book LMs | Natural Questions · Questions with few relevant documents | accuracy | Mixed |
Closest papers by meaning
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.92)
- Understanding Finetuning for Factual Knowledge Extraction (ICML 2024 · similarity 0.88)
- Long-form factuality in large language models (NeurIPS 2024 · similarity 0.88)
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.88)
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.88)
- Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.