Research map / Large language models / Language, safety and interpretability

Long-Short Alignment for Effective Long-Context Modeling in LLMs

ICML 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes long-short misalignment, a symmetric cross-entropy metric between output distributions on suffixes of different lengths, and uses it as a regularizer to improve length generalization when fine-tuning Llama2-7B with CLEX, LongQLoRA, and EABF.

Problem

Length generalization in LLMs is poor: models trained on short contexts degrade on longer test inputs, and prior work focuses on positional encodings and mechanisms rather than the consistency of output distributions across lengths.

Approach

Shift focus from input features to output-space consistency: penalize divergence (symmetric cross-entropy) between model predictions on suffixes of different lengths of the same sequence, and add this long-short misalignment as a regularization term during training, implemented efficiently by reusing overlapping positions of one longer sequence.

Key results

• Long-short misalignment correlates with LongBench-E score at -0.85 versus -0.55 for training loss across seven models.
• Adding 0.1*L_misalign to CLEX fine-tuning on RedPajama-Book raised LongBench-E from 24.7 to 26.6 at 200 steps, with perplexity 5.88 versus 6.12.
• On BABILong at 8K and 16K, the method gains 2.0% and 2.2% over baseline (CLEX, 200 steps).

Measured effects reported in the paper

Surprising findings

• Encouraging alignment between sequences with large length differences (sampling l_extra over the full training range) severely degrades performance, with LongBench-E dropping to 18.2 at alpha=0.1 setting 4.
• Overly large regularization coefficient (alpha=1.0) causes perplexity to rise sharply (e.g., 16.21 vs 6.12 baseline), indicating over-regularization harms next-token prediction.

Limitations

• Causal relationship between misalignment and length generalization is not established; only correlation is shown.
• Performance is sensitive to the regularization coefficient and sampling range; large values hurt perplexity and benchmark scores.
• Evaluation is limited to Llama2-7B with a few adaptation methods and small training budgets (200 steps).

Experimental setup

Datasets: RedPajama-Book PG19 LongBench-E BABILong

Benchmarks: LongBench-E BABILong perplexity on 8k/16k validation sets

Metrics: LongBench-E score perplexity BABILong accuracy correlation coefficient with long-context performance

Baselines: CLEX baseline (cross-entropy only) LongQLoRA baseline EABF baseline RandomPos Yarn-Llama-2-7B-8k Qwen-7B-8k

Builds on: CLEX LongQLoRA EABF LongLoRA Position Interpolation

Closest papers by meaning

• The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.91)
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.91)
• Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs (NeurIPS 2025 · similarity 0.90)
• LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs (ICLR 2025 · similarity 0.90)
• Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.90)
• Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.90)