Research map / Large language models / Language, safety and interpretability
Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Quest, a query-centric data synthesis method that uses doc2query-predicted queries and keyword-based inverted indexes to concatenate semantically relevant yet diverse documents for long-context continued pretraining.
Problem
Long-context continued training relies on filtered long documents that are domain-imbalanced, while existing concatenation methods either sacrifice semantic coherence (random concatenation) or diversity (similarity-based KNN/ICLM).
Approach
Predict multiple queries per document with a doc2query model, extract keywords from those queries to index documents, then sample and concatenate documents sharing a keyword, with oversampling of rare keywords to balance the data distribution.
Key results
- Quest-LLaMA-3-8B-128k scores 32.39 on Longbook QA, the best among open-source models and above LLaMA-3.1-8B-base (30.11).
- On Longbench at 32K, Quest reaches 22.06 average for 1.4B vs 20.94 for Standard, and 25.24 for 12B vs 24.85 for Standard.
- Quest-LLaMA-3-8B reaches 100% accuracy on Needle-in-a-Haystack up to 1M context length (numeric string retrieval) and 97% at 128K for sentence retrieval.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use query-centric Quest synthesis instead of Standard random concatenation | Pythia continued pretraining · Longbench 32k Avg. | score: 20.94 -> 22.06 | Helped |
| Use query-centric Quest synthesis instead of KNN retrieval concatenation | Pythia continued pretraining · Longbook QA 128k | score: 16.42 -> 18.92 | Helped |
| Use Quest-synthesized long-context data instead of existing long documents | Pythia-1.4B · Longbench 32k Avg. | score: 21.11 -> 22.06 | Helped |
| Add query-based synthesis to LLaMA-3-8B | LLaMA-3-8B · Needle-in-a-Haystack 128k | accuracy: 88% -> 97% | Helped |
| Replace query keywords with randomly selected keywords at higher ratios (ablation) | Quest · HELMET Avg. (8k–128k) | score: 50.97 -> 50.33 -> 49.74 -> 48.08 | Hurt |
| Use keywords from queries rather than keywords from summaries (ablation) | Quest keyword extraction · Human/GPT-4o keyword quality (100 samples) | average score: 3.009 -> 3.574 | Helped |
| Randomly sample eligible keywords instead of choosing the highest RAKE score (ablation) | Quest · HELMET Avg. (8k–128k) | score: 49.51 -> 50.97 | Helped |
| Randomly sample eligible keywords instead of choosing the highest RAKE score (ablation) | Quest keyword selection · Keyword entropy | entropy score: 15.31 -> 18.12 | Helped |
| Increase split_ratio from 0% to 10% (ablation) | Quest oversampling · Longbench trec | score: 33.46 -> 39.21 | Helped |
| Increase split_ratio from 10% to 70% (ablation) | Quest oversampling · Longbench lcc | score: 33.96 -> 29.78 | Hurt |
| Use KNN mid-ranking documents rather than top-similarity documents (ablation) | KNN data synthesis · HELMET Avg. | score: 46.31 -> 47.96 | Helped |
| Use KNN random sampling rather than top-similarity documents (ablation) | KNN data synthesis · HELMET Avg. | score: 46.31 -> 46.78 | Mixed |
Surprising findings
- Moderately similar KNN-aggregated documents (mid-ranking) did not match Quest's performance, suggesting document aggregation matters more than similarity level.
- Standard random concatenation performs best on some 32K tasks and is weakest on 128K Longbook QA, contrary to expectations of diversity-first benefits.
Limitations
- Query prediction uses a lightweight doc2query model rather than LLM-generated queries due to compute cost; the authors note higher-quality queries could help.
- Experiments are mainly on Pythia models and the Pile; ablations are limited to 1.4B at 32k.
- Long documents above 128k are rare in the Pile, so the 128k comparison against existing long documents could not be done.
Experimental setup
Datasets: The Pile Longbench Longbook QA HELMET WinoGrande PIQA LogiQA LAMBADA HellaSwag ARC-Easy ARC-Challenge Needle-in-a-Haystack
Benchmarks: Longbench Longbook QA HELMET Needle-in-a-Haystack WinoGrande PIQA LogiQA LAMBADA HellaSwag ARC-Easy ARC-Challenge
Metrics: accuracy F1 Levenshtein similarity average score entropy
Baselines: Standard KNN ICLM Long document
Builds on: doc2query in-context pretraining retrieval-augmented language model pretraining RAKE
Closest papers by meaning
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.89)
- Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.88)
- Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data (ICLR 2025 · similarity 0.87)
- NExtLong: Toward Effective Long-Context Training without Long Documents (ICML 2025 · similarity 0.87)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.87)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.