Research map / Large language models / Instruction and fine-tuning
Rote Learning Considered Useful: Generalizing over Memorized Data in LLMs
Qinyuan Wu, Soumi Das, Mahsa Amani, Bishwamittra Ghosh, Mohammad Aflah Khan, Krishna P. Gummadi, Muhammad Bilal Zafar
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase rote memorization epochs in Phase-1 (3 to 20) (ablation) | memorize-then-generalize · Synthetic author facts, unseen prompts | test prompt accuracy: 0.35 -> 0.98 | Improved |
| add Phase-2 fine-tuning with one meaningful prompt on 50 memorized facts | rote-memorized Qwen2.5-1.5B · Synthetic facts, unseen facts | generation accuracy: 0.36 -> 0.76 | Improved |
| add Phase-2 fine-tuning on held-out facts probability | rote-memorized Qwen2.5-1.5B · Synthetic facts, train prompt on held-out facts | object probability: 0.18 -> 0.79 | Improved |
| use a single memorized fact with a single prompt in Phase-2 (ablation) | memorize-then-generalize · Synthetic author facts, test prompts | accuracy: 0.76 (k=1) | Improved |
| rote memorization of the first relation before learning the second (ablation) | memorize-then-generalize · 2-hop composition A->C | generation accuracy: 0.14 -> 0.36 | Improved |
| 5 to 15 epochs of rote memorization of R1 instead of 20 (ablation) | memorize-then-generalize · 2-hop composition A->C | generation accuracy: 0.14 -> 0.14 | No clear effect |
| add rote memorization before reversal training | memorize-then-generalize · Synthetic mother relation, reversal | accuracy: 0 -> 0.26 | Improved |
| use memorize-then-generalize instead of SFT on reversal | Qwen2.5-1.5B · Synthetic mother relation, reversal | accuracy: 0.01 -> 0.26 | Improved |
| use memorize-then-generalize instead of SFT at equal training tokens | Qwen2.5-1.5B · Synthetic facts, one training prompt | accuracy | Improved |
| use memorize-then-generalize instead of SFT with ten training prompts | Qwen2.5-1.5B · Synthetic facts, ten training prompts | accuracy and training tokens: both ~0.9 accuracy, half the tokens | Mixed |
| memorize-then-generalize in place of ICL for knowledge injection | Qwen2.5-1.5B · Synthetic facts | object probability | Improved |
| memorize subject-object pairs without the key token (no semantic anchor) (ablation) | memorize-then-generalize · Synthetic facts | generalization | Hurt |
Closest papers by meaning
- Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.90)
- Mitigating Memorization in Language Models (ICLR 2025 · similarity 0.88)
- Quantifying and Analyzing Entity-Level Memorization in Large Language Models (AAAI 2024 · similarity 0.87)
- Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.86)
- What Do Learning Dynamics Reveal About Generalization in LLM Mathematical Reasoning? (ICML 2025 · similarity 0.86)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
