Research map / Large language models / Language, safety and interpretability

LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LongMagpie, which synthesizes long-context instruction data by prompting aligned LLMs with a document followed by a user-role pre-query token, plus a p-Mix strategy for balancing long and short training data.

Problem

Long-context instruction data for aligning long-context LLMs is proprietary or costly to create; human annotation is expensive and template/seed-based synthesis limits scale, diversity, and quality.

Approach

Aligned long-context LLMs, given only a document followed by the special tokens preceding a user turn, auto-regressively generate relevant queries; harvesting these document-query pairs and the model's responses yields instruction data without seeds or prompts. Extends to multi-document contexts and mixes long and short instructions via probabilistic sampling (p-Mix).

Key results

• LongMagpie-only training reaches LongAVG 62.56 vs 60.28 for ChatQA and 56.12 for LongAlign, with LongBench v2 at 34.4.
• LongMagpie + UltraChat via p-Mix achieves LongAVG 61.60 and ShortAVG 64.10, retaining short-context performance.
• Scaling from 190k to 450k samples raises LongAVG from 61.51 to 62.56; using Qwen2.5-70B instead of 7B raises LongAVG from 59.61 to 62.56.

Measured effects reported in the paper

Surprising findings

• Multi-document extension with up to n=10 documents works best; performance declines at larger n, attributed to excessive task difficulty.
• LongMagpie-only training gives the best long-context scores but a lower ShortAVG (62.37) than short-only datasets such as UltraChat (64.43).

Limitations

• Inherits biases from the source instruction-tuned LLMs.
• Does not adequately cover tasks requiring long-form outputs.
• Query filtering is heuristic (question mark and length rules).

Experimental setup

Datasets: FineWeb-Edu LongMagpie-generated instructions ChatQA2 data LongAlign Tulu Magpie UltraChat

Benchmarks: HELMET RULER LongBench v2 HellaSwag Lambada ARC-Challenge ARC-Easy PIQA WinoGrande LogiQA

Metrics: accuracy LongAVG ShortAVG

Baselines: Tulu Magpie UltraChat ChatQA LongAlign Sequential Mix Simple Mix

Builds on: Magpie NExtLong ProLong

Closest papers by meaning

• Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.92)
• LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.92)
• MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.90)
• LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs (ICLR 2025 · similarity 0.89)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.89)
• LOGO --- Long cOntext aliGnment via efficient preference Optimization (ICML 2025 · similarity 0.89)