Research map / Large language models / Instruction and fine-tuning
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LongLoRA, which extends Llama2 context to 32k-100k with shifted sparse attention during fine-tuning plus LoRA with trainable embedding and normalization layers, on a single 8xA100 machine.
Problem
Extending the context window of pre-trained LLMs via fine-tuning is computationally expensive because self-attention cost grows quadratically, and plain LoRA performs poorly for long-context adaptation.
Approach
Shifted sparse attention (S^2-Attn) splits tokens into groups for local attention during training, shifting half the heads by half a group so information flows between groups; standard full attention is kept at inference. Combined with LoRA that also trains embedding and normalization layers, this closes the gap to full fine-tuning.
Key results
- LongLoRA with S^2-Attn and LoRA+ reaches 2.72 PPL on PG19 at 8192 for Llama2 7B, close to full fine-tuning's 8.08 PPL at 8192 setting range (full-attention baseline 2.66 on proof-pile at 8192 eval).
- Llama2 7B extended to 100k context on a single 8xA100 machine with perplexity 2.52 on proof-pile; Llama2 70B extended to 32k.
- Training hours at 65536 context: LongLoRA 52.4 h vs LoRA 92.5 h; plain LoRA at rank 8 gives PPL 11.44 vs full FT 8.08 at 32k, while adding trainable norm and embed gives 8.12.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add S²-Attn instead of short attention without shifting | Llama2-7B long-context fine-tuning · PG19 validation, target context 8192 | perplexity: 8.29 -> 8.04 | Helped |
| Train normalization and embedding layers alongside LoRA rank 8 (ablation) | Llama2-7B · PG19 validation, context 32768 | perplexity: 11.44 -> 8.12 | Helped |
| Train normalization layers alongside LoRA rank 8 (ablation) | Llama2-7B · PG19 validation, context 32768 | perplexity: 11.44 -> 10.49 | Helped |
| Train embedding layers alongside LoRA rank 8 (ablation) | Llama2-7B · PG19 validation, context 32768 | perplexity: 11.44 -> 8.29 | Helped |
| Increase LoRA rank from 8 to 256 while keeping embeddings and normalization frozen (ablation) | Llama2-7B · PG19 validation, context 32768 | perplexity: 11.44 -> 11.98 | Hurt |
| Reverse the S²-Attn shifting direction (ablation) | Llama2-7B fine-tuned with S²-Attn · PG19 validation, context 8192 | perplexity: 8.04 -> 8.04 | No clear effect |
| Put shifted tokens in individual groups (ablation) | Llama2-7B fine-tuned with S²-Attn · PG19 validation, context 8192 | perplexity: 8.04 -> 8.03 | No clear effect |
| Swap shifted tokens with original front tokens (ablation) | Llama2-7B fine-tuned with S²-Attn · PG19 validation, context 8192 | perplexity: 8.04 -> 8.05 | No clear effect |
| Reduce S²-Attn group size to 1/8 of target context (ablation) | Llama2-7B LongLoRA · PG19 validation, context 8192 | perplexity: 8.04 -> 8.16 | Hurt |
| Shift between sequential layers rather than attention heads (ablation) | Llama2-7B long-context fine-tuning · PG19 validation, context 32768 | perplexity: 8.12 -> 9.70 | Hurt |
| Use pattern 1 in every attention head instead of mixing patterns (ablation) | Llama2-7B long-context fine-tuning · PG19 validation, context 32768 | perplexity: 8.12 -> 8.39 | Hurt |
| Replace S²-Attn with dilated attention (ablation) | Llama2-7B long-context fine-tuning · PG19 validation, context 32768 | perplexity: 8.12 -> 11.78 | Hurt |
Surprising findings
- Increasing LoRA rank up to 256 does not close the long-context gap to full fine-tuning; trainable normalization and embedding layers are key.
- Shifting between sequential layers is acceptable but worse than shifting among heads, and dilated and stride sparse attention patterns from training-from-scratch transformers perform poorly when fine-tuning pre-trained LLMs.
Limitations
- Small context-size perplexity degradation for extended models, a known limitation of Position Interpolation.
- Evaluated mainly on Llama2 models; authors state generality to other LLMs and position encodings is future work.
- Passkey retrieval accuracy degrades beyond the fine-tuned context length (about 33k-34k for 32k training).
Experimental setup
Datasets: RedPajama PG19 proof-pile LongAlpaca-12k LongBench LEval LongChat topic retrieval
Benchmarks: PG19 proof-pile LongChat topic retrieval passkey retrieval LongBench LEval
Metrics: perplexity accuracy win-rate
Baselines: Full fine-tuning LoRA Position Interpolation LongChat-13B LongChat-v1.5-7B Vicuna-v1.5-7B GPT-3.5-Turbo Landmark attention
Builds on: LoRA Position Interpolation Flash-Attention2 Swin Transformer
Closest papers by meaning
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.91)
- VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections (NeurIPS 2024 · similarity 0.91)
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.90)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
- Training-Free Long-Context Scaling of Large Language Models (ICML 2024 · similarity 0.90)
- Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.