Research map / Architectures and efficiency / Attention and transformers

Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Jet-Long, a tuning-free zero-shot context extension method that pairs a protected local RoPE window with a remote window whose analytic dynamic group size G=ceil(L/w_pretrained) aliases positions, with an inclusion-exclusion FlashAttention-based fused implementation.

Problem

Zero-shot context extension of RoPE-based LLMs beyond their pretrained window: fixed rescaling factors trade short-context fidelity against long-context reach, and length-adaptive methods rely on fitted schedules and incur inference overhead.

Approach

A bifocal (local + remote) attention where the remote group size is derived analytically from the current sequence length as the minimal integer compression keeping remote relative positions in the pretrained range; cache keys are kept at base positions and corrected on the fly with a RoPE correction rotation, and the two-window attention is computed via an inclusion-exclusion LSE merge in a fused

Key results

• On Qwen3-1.7B/4B/8B-Base up to 128K, Jet-Long leads the strongest baseline on RULER by +4.79/+2.18/+2.03 pp (RULER average 81.23 across sizes).
• On H100 at 64K–128K, fused Jet-Long prefill reaches 0.83–0.88x FlashAttention-3 throughput and 0.88–0.93x a matched CuTe control.
• On Jet-Nemotron-2B, Jet-Long raises 128K RULER accuracy from 8.54 to 33.78.

Surprising findings

• Unfused Jet-Long decode runs at only 0.15–0.21x FA3 throughput, so the fused kernel is essential for practicality.
• Aliasing-based position mapping underperforms frequency interpolation on some subtasks (e.g., MK-NIAH-2/3, CWE, QA-2 at 128K).

Limitations

• Retains dense quadratic prefill and linear KV cache; no matched comparison with training-based methods like LongRoPE.
• Cache correction does not recompute historical hidden states, so retained-cache decoding is not equivalent to full re-prefill.
• Evaluation limited mainly to RULER, HELMET-RAG and PG-19; error bars reflect dataset sampling, not training variance.

Experimental setup

Datasets: RULER HELMET-RAG PG-19

Benchmarks: RULER HELMET-RAG PG-19

Metrics: accuracy perplexity throughput (tok/s)

Baselines: DNTK YaRN DCA Self-Extend Unextended base model

Builds on: Self-Extend Dual Chunk Attention FlashAttention NTK-aware scaled RoPE YaRN

Closest papers by meaning

• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
• LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning (ICML 2024 · similarity 0.89)
• Let's (not) just put things in Context: Test-time Training for Long-context LLMs (ICLR 2026 · similarity 0.89)
• PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.88)
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.88)
• Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.88)