Research map / Large language models / Language, safety and interpretability
Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
Zichong Li, Chen Liang, Liliang Ren, Tuo Zhao, yelong shen, Weizhu Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RoPE-perturbed self-distillation (skip) | Llama-3-8B-Instruct long-context SFT · RULER 64K | accuracy (%): 57.25 -> 69.29 | Improved |
| add RoPE-perturbed self-distillation (skip) | Qwen-3-4B · RULER 256K | accuracy (%): 67.01 -> 68.10 | Improved |
| use reverse KL instead of forward KL (ablation) | self-distillation objective · RULER 64K (Llama) | accuracy (%): 57.8 -> 59.2 | Improved |
| use KL distillation instead of CLM on perturbed view (ablation) | self-distillation objective · RULER 64K (Llama) | accuracy (%): 56.5 -> 59.2 | Improved |
| sample random skip (s,y) instead of a fixed skip (ablation) | self-distillation objective · RULER 64K (Llama) | accuracy (%): 55.1 -> 59.2 | Improved |
| add dropout-based two-pass consistency instead of RoPE perturbation (ablation) | generic regularizer · RULER 64K (Llama) | accuracy (%): 47.9 -> 48.9 | No clear effect |
| add Gaussian attention noise consistency (ablation) | generic regularizer · RULER 64K (Llama) | accuracy (%): 47.9 -> 49.6 | No clear effect |
| remove positional encoding in perturbed view (NoPE) (ablation) | self-distillation objective · RULER 64K (Llama) | accuracy (%): 59.2 -> 29.5 | Hurt |
| use chunked permutation of index blocks (ablation) | self-distillation objective · RULER 64K (Llama) | accuracy (%): 47.9 -> 46.2 | Hurt |
| use random dilation of RoPE indices (ablation) | self-distillation objective · RULER 64K (Llama) | accuracy (%): 47.9 -> 53.4 | Improved |
| lower KL coefficient lambda to 0.2 (ablation) | self-distillation objective · RULER 64K (Llama) | accuracy (%): lambda 0.2: 54.0; 0.5: 60.0; 1: 59.2; 2: 58.9 | Mixed |
| combine skip self-distillation with LongCE | Qwen-3-4B after SFT · LongBench-v2 | accuracy (%): 29.40 -> 32.22 | Improved |
Closest papers by meaning
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
- LongRoPE2: Near-Lossless LLM Context Window Scaling (ICML 2025 · similarity 0.90)
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.90)
- Doc-to-LoRA: Learning to Instantly Internalize Contexts (ICML 2026 · similarity 0.90)
- Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.89)
- Mixture of In-Context Experts Enhance LLMs' Long Context Awareness (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
