Research map / Large language models / Language, safety and interpretability

From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Finetunes GPT-3.5 Turbo and Mistral 7B on synthetic numerical key-value retrieval tasks, showing improved long-context retrieval (MDQA) and reasoning (FLenQA) with little general-benchmark degradation.

Problem

LLMs struggle to retrieve information and reason over long-context inputs (lost-in-the-middle, needle-in-a-haystack effects), and finetuning on long-context data may induce hallucinations.

Approach

Finetune LLMs on purely synthetic, fact-free numerical key-value dictionary retrieval tasks (with answer templates to focus loss on the value) so the retrieval skill transfers to real long-context QA and reasoning without harming general benchmarks.

Key results

• GPT-3.5 Turbo finetuned on synthetic key-value data gains 10.5% on 20-document MDQA at position 10 (abstract).
• Mistral 7B finetuned on synthetic data shows no TriviaQA drop, while other long-context baselines drop 2.33% to 6.19%.
• General benchmarks (MMLU, HellaSwag, GSM8K, TriviaQA, NQ-Open) remain nearly unchanged after synthetic finetuning (changes mostly within about ±0.5 points).

Measured effects reported in the paper

Surprising findings

• Finetuning on the synthetic key-value data without an answer template degrades chain-of-thought FLenQA performance for GPT-3.5 Turbo, while the templated version improves it.
• Synthetic finetuning does not help when MDQA distractors are relevant (retrieved) documents rather than irrelevant ones.

Limitations

• No improvement on MDQA with relevant distractors.
• Only two base models (GPT-3.5 Turbo and Mistral 7B) with limited finetuning scale; GPT-3.5 evaluated on fewer general benchmarks (MMLU on 20% subset).
• Synthetic data is purely numerical and may not cover broader instruction-following; proposed mixing with general instruction data as future work.

Experimental setup

Datasets: synthetic key-value retrieval (new) MDQA FLenQA MultidocQA IN2 Needle-in-a-haystack MMLU HellaSwag GSM8K TriviaQA NQ-Open

Benchmarks: MDQA FLenQA MMLU HellaSwag GSM8K TriviaQA NQ-Open

Metrics: accuracy maximum subspan exact match

Baselines: MultidocQA finetuning IN2 finetuning Needle-in-a-haystack finetuning MDQA-finetuned GPT-3.5 Turbo original GPT-3.5 Turbo original Mistral 7B

Builds on: lost-in-the-middle needle-in-a-haystack FLenQA

Closest papers by meaning

• Understanding Synthetic Context Extension via Retrieval Heads (ICML 2025 · similarity 0.91)
• Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.89)
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.88)
• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.88)
• A Controllable Examination for Long-Context Language Models (NeurIPS 2025 · similarity 0.88)