Research map / Large language models / Retrieval and knowledge
Learning Facts at Scale with Active Reading
Jessy Lin, Vincent-Pierre Berges, Xilun Chen, Wen-tau Yih, Gargi Ghosh, Barlas Oguz
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Active Reading data instead of synthetic QA (ablation) | Llama 3.1 8B Base finetuning · SimpleWikiQA | accuracy: 47.87 -> 66.25 | Improved |
| use Active Reading (task-agnostic) instead of paraphrasing (ablation) | Llama 3.1 8B Base finetuning · FinanceBench (info. extraction) | accuracy: 43.87 -> 66.18 | Improved |
| make Active Reading task-specific instead of task-agnostic (ablation) | Llama 3.1 8B Base finetuning · SimpleWikiQA vs FinanceBench info. | accuracy: 63.33 -> 66.25 (SimpleWikiQA); 66.18 -> 61.49 (FinanceBench info.) | Mixed |
| combine paraphrase, synth QA and Active Reading data (ablation) | Llama 3.1 8B Base finetuning · SimpleWikiQA | accuracy: 66.25 -> 66.66 | No clear effect |
| scale 4x and 16x more Wikipedia distractor documents in fine-tuning (ablation) | Active Reading finetuning · SimpleWikiQA | accuracy | Hurt |
| raise learning rate from 1e-5 to 3e-4 and mix in much more pretraining data (ablation) | Active Reading at scale · SimpleWikiQA and NaturalQuestions | accuracy | Improved |
| train 8B model on 70B-generated data instead of self-generated data (ablation) | Llama 8B Base, task-specific Active Reading · SimpleQA | accuracy: 66.25 -> 62.26 | Hurt |
| train 70B model on 70B-generated rather than 8B-generated data (ablation) | Llama 70B Base · SimpleQA | accuracy: 71.52 -> 77.14 | Improved |
| Active Reading finetune of 70B with 8B data vs no finetuning (ablation) | Llama 70B Base · SimpleQA | accuracy: 75.76 -> 71.52 | Hurt |
| scale Active Reading to 1T tokens of Wikipedia and train WikiExpert | Llama 3.1 8B · SimpleQA | accuracy: 7.3 -> 23.5 | Improved |
| scale Active Reading to 1T tokens of Wikipedia | Llama 3.1 8B · NaturalQuestions | accuracy: 29.0 -> 31.2 | Improved |
| scale Active Reading to 1T tokens of Wikipedia | Llama 3.1 8B · GSM8k guardrail | accuracy: 54.4 -> 48.6 | Hurt |
Closest papers by meaning
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.86)
- Efficient Evaluation of LLM Performance with Statistical Guarantees (NeurIPS 2026 · similarity 0.86)
- Let's (not) just put things in Context: Test-time Training for Long-context LLMs (ICLR 2026 · similarity 0.85)
- Understanding Finetuning for Factual Knowledge Extraction (ICML 2024 · similarity 0.85)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.85)
- How Do Large Language Models Acquire Factual Knowledge During Pretraining? (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
