Research map / Large language models / Retrieval and knowledge
Make Your LLM Fully Utilize the Context
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes In2 training, a purely data-driven fine-tuning on synthesized long-context QA (4K–32K) that requires fine-grained awareness of 128-token segments and multi-segment reasoning, producing FilM-7B from Mistral-7B-Instruct-v0.2 to mitigate lost-in-the-middle.
Problem
Long-context LLMs fail to fully use information in the middle of long inputs (the lost-in-the-middle problem), which standard long-context training does not explicitly address.
Approach
Hypothesizes the bias comes from general training data placing important information at the beginning/end; synthesizes long contexts by shuffling many random 128-token segments and generating QA pairs with GPT-4-Turbo that require info at random positions, so any position can hold crucial information.
Key results
- FilM-7B reaches 85.9 average and 13.9 gap on VaL Probing overall, versus 79.0 average and 32.1 gap for GPT-4-Turbo and 47.3/56.2 for the Mistral-7B-Instruct-v0.2 backbone.
- On real-world LongBench tasks, FilM-7B averages 39.9 F1/ROUGE-L, above GPT-4-Turbo (44.7 avg is higher; FilM is the best ~7B open model), e.g. NarrativeQA 23.5 to 26.9 over the backbone.
- Short-context performance is maintained: MMLU 59.3 to 59.2 relative to the Mistral backbone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply In2 training | Mistral-7B-Instruct-v0.2 · VaL Probing All | Avg: 47.3 -> 85.9 | Helped |
| apply In2 training | Mistral-7B-Instruct-v0.2 · VaL Probing All | Gap: 56.2 -> 13.9 | Helped |
| apply In2 training | Mistral-7B-Instruct-v0.2 · NarrativeQA | F1: 23.5 -> 26.9 | Helped |
| apply In2 training | Mistral-7B-Instruct-v0.2 · Short-context tasks | accuracy: 59.3→59.2 | No clear effect |
| increase RoPE base from 1.0×10^6 to 1.0×10^7 during In2 training (ablation) | FilM-7B · VaL Probing All (20% training data) | Avg: 80.3 -> 84.9 | Helped |
| use a 4K sliding window during training (ablation) | In2-trained Mistral models · VaL Probing | retrieval performance | Hurt |
Surprising findings
- Models with near-perfect Needle-in-the-Haystack scores (LongAlign, InternLM2) still show lost-in-the-middle on VaL Probing, suggesting NIAH overestimates long-context ability.
- Sliding-window attention greatly hurts long-distance retrieval even after In2 training.
Limitations
- Relies on GPT-4-Turbo to generate QA pairs, incurring API cost and dependence on a proprietary model.
- Training is expensive (~300 GPU days), so ablations were run on only 20% of the data.
- Evaluation focused on 7B scale and specific probing/real-world tasks; the paper's own probing tasks are constructed by the authors.
Experimental setup
Datasets: C4 realnewslike OpenOrca StarCoder Wikidata arXiv abstracts
Benchmarks: VaL Probing Needle-in-the-Haystack NarrativeQA Qasper MultiFieldQA HotpotQA 2WikiMultihopQA MuSiQue GovReport QMSum MultiNews MMLU
Metrics: average score min-max gap word-level recall exact-match accuracy F1 ROUGE-L accuracy
Baselines: Mistral-7B-Instruct-v0.1 Mistral-7B-Instruct-v0.2 LongAlign-7B-64K LongAlign-13B-64K InternLM2-chat-7B InternLM2-chat-20B GPT-4-Turbo GPT-3.5-Turbo LongChat-v1.5-7B-32K ChatGLM2-6B-32K ChatGLM3-6B-32K Yi-6B-200K
Builds on: Mistral-7B-Instruct-v0.2 LongBench Needle-in-the-Haystack OpenOrca
Closest papers by meaning
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.90)
- NExtLong: Toward Effective Long-Context Training without Long Documents (ICML 2025 · similarity 0.89)
- Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.89)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.89)
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.88)
- An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.