Research map / Large language models / Language, safety and interpretability

Mixture of In-Context Experts Enhance LLMs' Long Context Awareness

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MoICE, a plug-in router in each attention head that dynamically mixes multiple RoPE angles (treated as in-context experts) to improve LLM long-context awareness, trained router-only on frozen LLMs.

Problem

LLMs have uneven awareness of contextual positions under RoPE, so critical information in certain positions (e.g., the middle of long contexts) is overlooked, hurting long-context understanding, generation and RAG.

Approach

Treat each RoPE angle (base) as an in-context expert and attach a learned top-K router to every attention head that selects and weights K RoPE angles per token, aggregating their attention distributions; only the routers are trained with the LLM frozen.

Key results

• On L-Eval, MoICE on Llama2-7B-Chat raises the average from 47.52 to 50.72 and on Mistral-7B-Instruct-8k from 53.27 to 55.38, outperforming AB and Ms-PoE.
• On MDQA with Mistral-7B-Instruct-8k, MoICE reaches 53.30 average versus 50.65 for the base model and 50.81 for AB; on Llama2 the position gap drops to 2.22 points.
• Router-only training with MoICE improves Qwen1.5-7B-Chat L-Eval average from 67.76 to 68.54 while AB and Ms-PoE run out of memory on several tasks; MoICE memory is ~19-25 GB vs 62-79 GB for AB on Llama2.

Measured effects reported in the paper

Surprising findings

• Randomly weighted expert selection collapses performance (average 18.65 on Llama2-7B-chat), showing the router weights matter.
• Standard fine-tuning of the full model degrades performance (e.g., Mistral L-Eval average drops to 31.92), attributed to catastrophic forgetting.

Limitations

• Did not test pretraining larger models with MoICE due to limited compute.
• MoICE does not add inherent context awareness beyond exploiting existing capacity; full-parameter training may be needed.
• Open-ended evaluation depends on a GPT-4-Turbo judge and a small set of 181 questions.

Experimental setup

Datasets: OpenHermes-2.5 (1k longest subset) Airoboros Long-Alpaca LongAlign OpenWebText NaturalQuestions-Open (MDQA)

Benchmarks: L-Eval (Coursera, QuALITY, TOEFL, SFiction, open-ended) LongBench MDQA (multi-document QA) Key-Value Retrieval

Metrics: exact match accuracy win-rate average score accuracy gap across positions inference time GPU memory

Baselines: Positional Interpolation Dynamic NTK Ms-PoE Attention Buckets Sliding window attention Fine-tuning

Builds on: Attention Buckets Ms-PoE RoPE Mixture of Experts L-Eval

Closest papers by meaning

• Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.90)
• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
• Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.89)
• PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.89)
• Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.89)
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.89)