Research map / Large language models / Retrieval and knowledge

Adapting Large Language Models via Reading Comprehension

ICLR 2024 · Large language models › Retrieval and knowledge · approach: Explanatory and multiple-choice QA benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows continued pre-training on raw domain corpora injects domain knowledge but hurts question-answering prompting, and proposes reading-comprehension-style training that converts raw corpora into reading comprehension tasks to restore and improve it.

Problem

Continued pre-training on domain-specific raw corpora gives LLMs domain knowledge but drastically harms their ability to answer questions via prompting.

Approach

Inspired by human learning via reading comprehension (practice after reading), the method augments raw domain corpora with automatically generated comprehension tasks so the model learns domain knowledge and also how to use it for question answering.

Surprising findings

• Training on raw domain corpora drastically hurts the model's prompting ability for question answering despite adding domain knowledge.

Experimental setup

Baselines: raw-corpus continued pre-training

Builds on: continued pre-training

Closest papers by meaning

• BLADE: Enhancing Black-Box Large Language Models with Small Domain-Specific Models (AAAI 2025 · similarity 0.89)
• SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.87)
• Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis (AAAI 2025 · similarity 0.87)
• Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.87)
• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.86)
• Conditional Language Learning with Context (ICML 2024 · similarity 0.86)