Research map / Large language models / Language, safety and interpretability

Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks

AAAI 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fine-tunes LLMs on long-context QA samples whose answers begin by paraphrasing the relevant original text, aiming to strengthen retrieval and mitigate "lost in the middle".

Problem

LLMs with long context windows still underperform on short-dependency composite long-context tasks (retrieve-then-answer), showing "lost in the middle" even when they pass needle-in-a-haystack tests.

Approach

Construct training answers that first paraphrase/repeat the key original sentences from the context (with document index) before answering, so the fine-tuned model explicitly performs and activates retrieval; train with QLoRA on about 5k GPT-4-generated samples.

Key results

• On LongBench, Qwen2-7B-Instruct average rises from 56.77 to 58.96 and Llama3-8B-Chinese-Chat from 58.63 to 64.39 with the method, with synthetic (passage retrieval) gains such as 63.00 to 65.00 for Qwen2 and 80.75 to 97.75 for Llama3.
• On NaturalQuestions Multi-Doc-QA (20 docs), Qwen1.5-4B-Chat average accuracy improves from 47.20 to 51.68 with the method, versus 48.08 for SFT without paraphrasing.
• MMLU drops by at most about 2% after fine-tuning, which the authors call acceptable.

Measured effects reported in the paper

Surprising findings

• Switching data generation from gpt-3.5-turbo to gpt-4 gave little overall improvement, suggesting the method does not need a very strong generator.
• Chain-of-thought-style prompting did not improve Multi-Document-QA accuracy for the base Qwen1.5-4B-Chat (46.48 vs 47.20 default).

Limitations

• Improvements on single-document-QA are unstable across models.
• The method targets short-dependency tasks; long-dependency tasks were not the focus.
• Evaluation relies on a small subset (200 samples per sub-dataset) of LongBench.

Experimental setup

Datasets: CommonCrawl Wudao 200GB Chinese corpus Arxiv CNKI Alpaca Llama2-chinese dataset

Benchmarks: LongBench NaturalQuestions Multi-Document-QA MMLU

Metrics: accuracy rouge-L exact match

Baselines: Ziya-Reader SFT w/o ours Qwen1.5-4B-Chat Qwen2-7B-Instruct Llama3-8B-Chinese-Chat

Builds on: LongAlpaca Ziya-Reader QLoRA Lost in the middle

Closest papers by meaning

• Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.93)
• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.91)
• Make Your LLM Fully Utilize the Context (NeurIPS 2024 · similarity 0.90)
• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
• Structured Packing in LLM Training Improves Long Context Utilization (AAAI 2025 · similarity 0.90)
• NExtLong: Toward Effective Long-Context Training without Long Documents (ICML 2025 · similarity 0.90)