Research map / Large language models / Retrieval and knowledge
Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models
Yukun Huang, Sanxing Chen, Jian Pei, Manzil Zaheer, Bhuwan Dhingra
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Active Indexing instead of Passive Indexing | Qwen2.5-7B · RepliQA | citation precision (%): 2.4 -> 24.4 | Improved |
| add Active Indexing instead of Passive Indexing | Qwen2.5-7B · SciQAG | citation precision (%): 2.4 -> 32.6 | Improved |
| add Active Indexing instead of Passive Indexing | Qwen2.5-7B · ASQA | citation precision (%): 24.1 -> 30.9 | Improved |
| add backward augmentation (ablation) | Active Indexing · RepliQA | citation precision (%): 12.6 -> 24.4 | Improved |
| add forward augmentation to backward (ablation) | Active Indexing · RepliQA | citation precision (%): 21.6 -> 24.4 | Improved |
| append document IDs after each fact (Repeat) (ablation) | Passive Indexing · ASQA | citation precision (%): 24.1 -> 20.5 | Hurt |
| scale model size 3B to 14B under Active Indexing | Qwen2.5 ActIdx · RepliQA | citation precision (%): 10.5 -> 30.8 | Improved |
| replay passive indexing for more epochs (ablation) | Passive Indexing · RepliQA ModelQA | Acc@1 (%): 7.8 -> 6.0 | Hurt |
| add hybrid joint internal+external citation under noisy retrieval (0.2) | Active Indexing · RepliQA | accuracy (%): 19.8 -> 36.1 | Improved |
| add internal citation fallback when retrieval is dense (1.0) | Hybrid-Fallback vs External-only · RepliQA | accuracy (%): 62.2 -> 62.5 | No clear effect |
| use Active Indexing instead of passive title-ID CPT on OOD text (ablation) | Qwen2.5-7B CPT · Wikipedia perplexity | perplexity: 54.31 -> 26.52 | Improved |
| narrow RepliQA-only CPT vs base model on OOD TriviaQA | Qwen2.5-7B · TriviaQA | accuracy (%): 56.0 -> 50.5 | Hurt |
Closest papers by meaning
- Chain-of-Thought Improves Text Generation with Citations in Large Language Models (AAAI 2024 · similarity 0.88)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.88)
- ClashEval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence (NeurIPS 2024 · similarity 0.87)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.87)
- Synthetic continued pretraining (ICLR 2025 · similarity 0.87)
- CiteME: Can Language Models Accurately Cite Scientific Claims? (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
